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Flipping the Course Evaluation Process: 


Using Student Feedback Up Front (and Throughout) 


Steven M. Baule 
College of Education 
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Keywords: student evaluation of teaching (SET), student engagement 


Faculty are engaged in ensuring students learn. If students fail, instructors have failed. 
One way to ensure student success is to ask the students themselves. Asking at the beginning of 
the course is additionally a way to communicate to the students that the instructor wants them to 
be successful. Sometimes simply knowing that the professor cares can be a significant bump to 
student motivation and all they need to fuel their success for the rest of the term. Concern for and 
interest in the student is a key facet students expect in superior college teachers (Feldman, 1976). 
Asking for student feedback routinely during a course is an excellent way to show concern for 
student success and make modifications to improve student success. 


Project Questions 


Due to the continuation of the Pandemic enforced changes at the university level, and the 
clear direction for graduate programs to remain in online environments, it seemed essential to 
determine the best methods of obtaining student feedback to improve the student experiences in 
online courses. There were two primary questions to be answered: 


Question |: Did soliciting student input earlier than the end of course SET surveys result 
in higher student response rates? 


Question 2: Did initial and mid-course student feedback surveys result in actionable data 
to improve instruction? 


This SoTL project was limited to fully online courses as part of graduate educational 
leadership programs at two midwestern universities. 


Background 


As the cost of higher education is continuing to rise and universities are competing for a 
shrinking pool of students, it seems essential to ensure student success. Taking student feedback 
into consideration is a key element to designing effective instruction (Pate & Hunt, 2015). 
Frequent, timely, and constructive feedback is considered a key element of the American 
Association of Colleges and Universities’ (AAC&U) High Impact Practices (HIPs) (Lee, 
Wilkum, Immel, & Fisher, 2021). Fink (2005) includes feedback and assessment as one of the 
three core pillars of that instructional design model. Keiler, Diotti, Hudon, & Ransom (2020) 
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found that structured feedback from multiple sources including students could have a positive 
impact on improving teaching. Their study was a mentoring model with high school teachers, but 
the results should generalize to higher education instructors in a variety of input formats. 


Often, the traditional student evaluation of teaching (SET) at the end of the course is not 
taken seriously by the instructor and/or institutional stakeholders. According to Wong and Moni 
(2014), the evidence shows instructors were “moderately receptive” to student feedback. 
Research by Kember, Leung, & Kwan (2002) showed that in reviewing 25 sets of student 
feedback gathered by departments over three or four year periods, only one significant positive 
change was seen. Kember, Leung, & Kwan also found that the SET end of course evaluations 
were too focused on the teacher and not necessarily on the student experience within a course. In 
three other areas, student evaluations saw negative trends. Therefore, instructors and institutions 
do not seem to be gathering appropriate actionable feedback or they are not taking action based 
on SET data. (Wong & Moni, 2014). Both Hadad, Keren & Neveh (2020) and Deale (2020) 
question the overall validity of SET data and claim the validity of such end of course evaluations 
Deale articulated gender and age bias can also impact SET data. 


Wong & Moni (2014) identify that the purpose of SET is not necessarily clear. If it is 
primarily to improve the student experience, the timing at the end of the course is not conductive 
for the students completing the survey. As they have generally completed the course, instead of 
providing feedback to improve the student experience, they are more likely to be an evaluation of 
the instructor. Berezval, Lukats & Molontay (2021) identify that SET evaluations can be 
significantly manipulated based on such non-instructional things as passing out chocolate to 
students prior to completing the end of course evaluations. They state students’ anticipated or 
actual grade also has an impact on SET scores and grade inflation can play a part in raising 
instructor scores on SET evaluations. Steyn, Davies, & Sambo (2018) identify non-response 
error as a concern in SET evaluations as well since low response rates are common in end of 
course evaluations. However, Wurf & Povey (2020) found that students as young as those in 
primary grades can provide useful perceptions and feedback for instructors willing to integrate 
student feedback and consider student perspectives. 


Lee, Wilkum, Immel, & Fisher (2021) do not go far enough in their advocacy for 
feedback to better engage students. They focus on the instructor providing frequent, timely, and 
constructive feedback (p. 192) to the student, but they do not go to the next step and elicit 
feedback from the students about their perspectives about the course content, materials, and 
instructor. Mandouit’s (2017) study showed that gathering regular collaborative feedback from 
students and allowing instructors access to appropriate professional development and guided 
reflection can have a positive impact on instructional effectiveness. Pate & Hunt (2015) focused 
on the need to provide regular feedback to students in order to develop the necessary habitus to 
make use of the feedback being provided by the instructor. 
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Hortsch’s (2019) work identified that student feedback showed instructors that students 
used instructional materials and resources in unexpected ways and by gathering feedback, the 
instructors were able to utilize that information to provide more efficient instructional materials 
within the course. Gathering such information during the course itself would potentially allow an 
instructor to make modifications to the course in progress. Steyn, Davies, & Sambo (2018) 
pointed out that collaborative feedback opportunities generated slightly more recommendations 
than individual feedback opportunities. Awidi, Paynter, & Vujosevic (2019) found that students 
found social media outside the learning management system (LMS) such as Facebook a more 
engaging tool than the LMS’s discussion tools. However, using tools outside the LMS or the 
larger digital learning environment (DLE) seems to be unwise when addressing student 
perceptions of courses and instructors. 


Tormey, Hardebolle, Pinto, & Jermann, (2020) identified that the existing learning 
analytics in most LMSs are not yet aligned to any theoretical frameworks. In most cases, the 
analytics are simply off the shelf. Tormey, Hardebolle, Pinto, & Jermann identified that most 
LMS analytics are pragmatically based on available or east to obtain data and not currently 
aligned to pedagogical needs. 


Methodology 


According to Mandouit (2017), one of the most practical and effective methods of 
obtaining student feedback was to use a survey with both structured and open-ended questions. 
Based on that consideration, surveys were determined to be the best method for this project. 
Previously, the author used occasional mid-term surveys to gather input on a specific aspect of 
the course beyond the required SET end of course surveys. For instance, the following questions 
were included in a mid-term survey to an asynchronous accelerated graduate course: 


1. Do you feel you are learning enough in the course at this time? 
a. Yes (100%) 
b. No (0%) 
2. Do you think the instructor is participating or moderating the discussion questions? 
a. Too actively (not allowing enough student discussion) (0%) 
b. About the right amount (100%) 
c. Not enough interaction (0%) 
3. Would you prefer to have a couple of questions to select from or continue to have 
everyone in the class respond to the same questions? 
a. Choice would be good (48%) 
b. Keep everyone focused on the same questions (31%) 
c. It doesn’t matter to me (23%) 
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The parenthetical numbers show the results of that survey. Since the answer to the 
Question # 3 was the prime consideration in that survey, the students in the next iteration of the 
course were again asked if they preferred choice among the discussion questions (DQs) to which 
they were being asked to respond. In that case, the question was Do you prefer to have a couple 
of questions to select from in the discussion threads or would you prefer less choice in the 
discussion prompts? Ninety-four percent of the students responded choice is good and six 
percent said they didn’t have a preference. No one preferred to have everyone focus on a single 
set of DQs after they had experienced choice. After the success of these pilot surveys, the 
instructor decided to make feedback surveys a more regular part of the instructional process in 
most courses. 


Surveys 


This informal scholarship of teaching and learning (SoTL) project encouraged student 
feedback throughout five online graduate courses. The students were provided with an initial 
survey after the first synchronous class session for semester long synchronous courses (Courses 
A & B) to ask students what the instructor could do to assist them in being successful. Those 
students were given a mid-term survey to provide feedback to the instructor as well as the 
traditional end of course survey. For the students in accelerated (seven or eight week) 
asynchronous courses (Courses C, D & E), the students were given either an initial or a mid-term 
course survey and then a traditional end of course survey. Courses A, B, & C were taught at a 
university that did not require a SET end of course exam. Courses D & E were taught at a 
university where all students were sent a SET survey for each course. As all of the courses were 
taught in an online environment, the surveys were conducted online as well. The instructor 
posted a direct link to the survey in the announcement section of the learning management 
system a week before the end of the course as well as sending an institutional email to each 
student with a link to the survey and a request to provide constructive feedback to the instructor. 
The announcement generally followed the example below: 


Thanks again for all of your efforts this semester. I would like to ask you to take about 5 
minutes to complete a brief end-of-course evaluation. It will remain open until after I turn in 
grades. It is anonymous and you will not be asked for any identifying information. Of course, you 
can always provide feedback directly as well. URL included here. 


A reminder announcement was posted on the last day of the course following the example 
below: 


Don't forget the end of course evaluation available at URL included here. Thanks again 


for an excellent semester and let me know if you need anything. Best of luck going forward! 


The emails to each student followed a similar format. Approximately two-thirds of the responses 
were received prior to the reminder for the end of course surveys. 
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Collaborative Feedback via Discussion Threads 


Beyond the individual surveys, an “extra credit” or bonus discussion thread was added to 
the end of each course with discussion threads as part of the core coursework following the 
example below: 


Can you provide any suggestions for improving the course? 


Can you provide any suggestions to make any of the assignments or the syllabus clearer and 
easier to understand? Or remove any bias? 


Students were able to post and respond to the posts of other students to provide a collaborative 
opportunity for feedback beyond the individual opportunities provided through the surveys. 
Courses A, C, D, & E all included discussion thread requirements within the course, so a bonus 
thread was added to each as the example above to allow for a more collaborative feedback 
opportunity. 


Results 


Below in Table 1, the response rates for each survey are provided. The return rate of 
initial surveys was the highest. The initial survey response rate for Course A was due to the fact 
that a student dropped the course between the first class session and the official enrollment date. 
In all other cases, the enrollment of each course remained stable. For later calculations, Course 
A’s initial survey was considered to have a 100% return rate (see Table 1). 


Table 1 


Return Rates for Course Surveys 


Course A B C D E 
Initial Survey 105.3% | 69.0% | 83.3% | N/A N/A 
Mid-Term Survey 78.9% | 44.8% | N/A 81.3% | 72.2% 
End of Course 

Survey 84.2% | 65.5% | 66.7% | 18.8% | 33.3% 


Student response rates were much higher for the initial surveys than for the final surveys. Those 
students who were used to an institutional SET survey automatically being sent had the lowest 
response rates at an average of 26% in comparison to the return rate of 72% for the end of course 
surveys where they were not institutionally required. The completion rates of the initial feedback 
surveys were 81% on average with a range of 69% to 100%. 


The initial and mid-term surveys were not entirely uniform but were based upon specific 
instructional concerns. For instance, in one case, the survey was administered directly after the 
mid-term exam and two of the questions asked the students which of the study materials, they 
found useful, a review video and/or a study guide. This helped to guide the development of study 
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materials for future exams. The final question was generally an open question asking for 
suggestions to improve the course for the mid-term surveys. For the initial surveys, the final 
question was always What are three things I can do this semester to help build your confidence 
and ensure your success? 


The key phrases from that initial survey question to date have been to provide timely 
communications, constructive feedback and provide clear expectations and examples for 
assignments. Providing flexibility was often raised as a concern as well. Students also wanted to 
be able to revise assignments if they didn’t hit the mark on the first iteration. The student who 
asked for “gentle feedback” stands out as a particularly memorable response as instructors need 
to make sure that they provide constructive feedback but do not provide an overwhelming 
amount of feedback. As mentioned by McDonald, Rich, and Gubler (2019), occasionally 
feedback can border on the cruel or hurtful. The instructor generally tries to focus on no more 
than a couple of areas for improvement on any assignment. This allows students to build their 
confidence and does not overwhelm them early in the course. 


For comparison, the students in courses with a regular discussion thread as part of their 
course were given a bonus discussion thread as mentioned in the methodology. This allowed 
them to collaborative identify feedback for course improvement. The participation rate in those 
discussions ranged between 47% to 100% across the four courses. The median for the four 
courses was 83% participation in the end of course discussions (see Table 2). 


Table 2 


Participation in End of Course Feedback Discussion Thread 


Course A C D E 
Participants A7.A% 83.3% 100.0% 100.0% 
Average # of Posts per Respondent 2A 1.1 2.6 25 


There does not appear to be a clear reason for why Courses D & E had both higher participation 
rates and more active participation rates (shown by number of posts). It is possible, since those 
two courses had the lowest SET survey response rates, students felt more engaged with the 
collaborative discussion threads. 


A few of the students seemed to evidence a lack of comfort in sharing feedback in the 
discussion threads as if they were not able to voice their opinions freely. Though that is only 
conjecture. No follow up was conducted at the time of those comments. 
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Obtaining feedback from students during the course instead of waiting until the course 
was completed has been beneficial. Waiting until the traditional SET surveys at the end of the 
course may not engage students in part due to the fact any of their recommendations will not 
have a positive impact on their learning as the course in question is over. Initial and mid-term 
feedback surveys seemed to be a better way to engage more students. The response rate for 
surveys at the beginning of the course was much higher than those only offered at the end of the 
course. The midterm survey response rates were also higher on average than the end of survey 
results. 


The particularly low rate of survey responses for those end of course surveys where there 
was an institutional requirement for SET surveys may encourage those institutions to consider 
better data gathering methods. Particularly as the non-response loss mentioned by Steyn, Davies, 
& Sambo (2018) could be significant and lead to skewed results which are neither helpful for the 
institution nor the individual instructor. At the same time, one issue for institutions at the 
department and program levels is the lack of comparative data for instructors, particularly new 
instructors, to gauge their effectiveness in comparison with their peers. Some type of basic 
summary data to assist, at least non-tenured faculty, to be able to bring context to their SET data 
would be useful. 


Surveys should provide some level of open ended or unstructured prompts as well as the 
traditional Likert type items. McDonald, Rich, & Gubler (2019) point out that feedback is not 
always a good thing and in fact can be overly negative or even cruel at times in academic 
settings. The potential positives to improve the student experience should outweigh the potential 
negatives in most cases. For new faculty or those with thinner skins or other histories that might 
make them more susceptible to negative feedback, potentially, they could have a trusted peer 
filter the results and remove any non-constructive or cruel responses. This could provide an 
additional level of anonymity for students who might also feel comfortable with an additional 
level of filter 


Themes that emerged from the open-ended responses in the study surveys align to the 
fact students appreciate clear, timely and kind communications from instructors. Clear 
instructions on assignments, the opportunity to address areas of concern, obtain a review of an 
assignment prior to submission are all among the concepts raised by students. Students 
appreciate choice and prefer to feel some level of control over the instructional environment. 


Recommendations 
Recommendations for Further Study 


The study was conducted over two universities and a range of graduate courses including 
both aspiring K-12 administrators and traditional graduate students a few years or less out of 
undergraduate programs. Additionally, both traditional and accelerated length courses were 
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included along with both synchronous and asynchronous delivery methods. Those variables 
might account for some of the apparent spread in response rates. Similar studies should be 
conducted with more homogenous student populations including undergraduate populations as 
well as with more homogenous courses or multiple sections of the same course. Hujala, Knutas, 
Hynninen, & Arminen (2020) suggest using Latent Dirichlet Allocation (LDA) topic and cluster 
models to manage large scale open ended SET data and extract useful information for course 
improvement. Studies with large student populations should consider this option to balance allow 
for unstructured student responses even with those larger groups. 


Another potential untapped source of useful feedback are recent alumni. Contacting 
students after they have completed their programs of study may provide some of the best 
feedback about the practical viability of a course’s content. Another option would be to provide 
students the opportunity for feedback through an echo box or similar widget embedded within 
the course LMS or DLE to allow for ongoing anonymous student feedback. 


Recommendations for Practitioners 


Instructors should strongly consider initial course surveys to hear from students to learn 
more about their concerns and how to ensure student success. Mid-course surveys can also be 
helpful particularly after key assessments in order to obtain feedback on particular assignments 
and materials. Due to the particularly low return rates and the concerns raised about non- 
response bias by Steyn, Davies, & Sambo (2018) and others, institutions may wish to reconsider 
the high value that some put on SET data. It appears that there are better methods available for 
obtaining feedback to assist in the improvement of instruction. Developers and IT staff need to 
build out tools that can assist instructors in gathering more real time feedback. Tools to allow full 
analysis of discussion threads would be a key item in reviewing discussion based student 
feedback. 
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Introduction 


Looking for a full-time job in academia is challenging for both first-timers and those with 
multiple job hunt experiences due to the highly competitive job market. Especially, tenure-track 
faculty positions have always been competitive: Only 10% to 30% of Ph.D. holders get a tenure- 
track professor job (Effective Altruism Forum, 2020). The current COVID-19 pandemic and the 
declined economy make finding tenure-track faculty positions in educational or instructional 
technology fields harder than ever. The U.S. Department of Labor reported that state and local 
education employment dropped in both public and private sectors in almost all states due to 
funding cutbacks during the pandemic (Rosewicz & Maciag, 2020). As a result of the pandemic, 
faculty of all types, especially adjunct faculty, were faced with job losses or salary changes 
across all institutional classifications (Sziron, 2021). International students who need H1B and 
Green Card sponsorships face additional challenges in finding academic jobs at this time due to 
the increasing number of universities that decline the immigration sponsorship because of the 
financial difficulties as well as the political climate 


This article provides advice and reflections to those engaging in a tenure-track faculty job search 
in the field of educational and instructional technology. Although the paper's focus is geared 
more towards international students in America who seek tenure-track faculty positions, the 
advice provided will undoubtedly apply to other job-seeking students. The authors, tenured or 
tenure-track faculty members, and former international students in American universities bring 
together experiences looking for tenure-track positions in instructional technology and evaluating 
candidates for tenure-track positions as faculty members. The authors identified four strategic 
areas for consideration and discussion when students apply for tenure-track and other types of 
jobs in academia: reading job advertisements, creating documentation, preparing for interviews, 
and making online or campus presentations. Further, they discussed critical strategies and shared 
insights in managing the challenges of the job search. 
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Reading the Job Ads 


A careful reading of the job advertisement or job description is the first crucial step in job 
hunting. The job advertisement describes what kind of person the organization is looking for, 
including the required and preferred qualifications. An example of a typical required 
qualification in educational technology is a K-12 teaching experience due to a focus on K-12 
teacher education. Another significant type of program in the field focuses on corporate training. 
For such programs, work experiences in industry or the private sector could ensure getting a 
tenure-track job. If your capabilities are significantly different from the description in the job 
advertisement, this may not be an opportunity for you. Again, only 10% to 30% of Ph.D. holders 
get a tenure-track professor job (Effective Altruism Forum, 2020), which shows that the job 
market is highly competitive. If the job description does not make you feel "this sounds like me," 
then the chance of you getting the job will be slim. There are plenty of candidates in the market, 
and some candidates would certainly have qualifications that match the job descriptions. 


However, you should always apply for the job even if it does not entirely match your 
qualifications as long as you have some capabilities that match the job description. In many 
cases, you would find yourself meeting parts of the required and preferred qualifications. It is a 
difficult decision to make, choosing if you give up or try. However, due to the competitive job 
market of tenure-track positions, it is probably better to apply for more jobs than fewer. Another 
reason for the encouragement to apply is that the job description is about an ideal candidate. In 
reality, the institutes rarely find a perfect match in their searching efforts. Each candidate is 
unique, and so are you. You might have some qualifications that are not listed in the job 
descriptions but are helpful in the department. 


Once you decide to apply for the position, you want to make every effort to show your match to 
the advertised work. The Search Committee, who selects and recommends the candidate to hire, 
will evaluate your job application based on how close you are to the job description. You are not 
allowed to lie in your job application. You cannot list experiences that you do not have. 
However, you can strategically highlight your knowledge and expertise that are pertinent to the 
job description. 


Documentation: Letter, CV, and References 


As discussed in the previous section, showing your match to the job description is what you need 
to accomplish for your successful job hunt. The first step to show this match is to tailor your job 
documents to each position. In typical universities, tenure-track faculty members are evaluated 
based on their research, teaching, and service accomplishments, so you want to list and discuss 
your experiences in terms of the three areas. Research accomplishments include peer-reviewed 
journal publications and external and internal grant experiences. The best teaching experience 
would be your experience teaching a class as a solo teacher in the field of instructional design. 
However, other types of teaching experiences count, such as a solo teaching experience in 
another area or experience working as a teaching assistant in the field. Since the job market is 
very competitive, the key to your success is how many publications and teaching experiences 
you accumulate before the job hunt. While service experiences tend to be less emphasized than 
research and teaching experiences, you certainly want to have some service experiences inside 
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and outside your school. Services for academic organizations, such as Association for 
Educational Communications and Technology (AECT), will look strong in your job application. 
Such experiences at an academic organization will help your networking with faculty members 
in the field, too. The following sections discuss specific advice for each application material. 


Cover letters 


It would be best to describe what you have accomplished in your cover letter or letter of intent. 
Typically, the letter opened up a narrative about your interest in the position and followed by 
your academic credentials. A good cover letter should provide what the institution is looking for, 
and thus, it should vary according to the job descriptions. Suppose you are applying for a 
research-focused job. In that case, your cover letter will talk mainly about your research 
accomplishments, such as peer-reviewed journal publications and external and internal grants 
that you have received, and the relevant research experiences you have had. 


Meanwhile, suppose the position is more teaching-focused, such as a tenure-track position at a 
teaching institution or a clinical professor job in a research university. In that case, you want to 
spend more time talking about your teaching experience in the cover letter. While most tenure- 
track positions require strong qualifications both in research and teaching, some universities also 
emphasize service experience more. 


In general, your letter should highlight the required qualifications based on your past and current 
experiences. Also, you may try your best to address the preferred qualifications. For example, 
working as a research or teaching assistant could extend your presentation as long as possible, 
well-aligned with the preferred capabilities. 


In addition to the cover letters, some universities ask for research or service statements and 
teaching philosophies. The content of such a write-up would overlap with the cover letter 
content, but it would provide you with more space to discuss your experience. Many books and 
online articles are available on how to write a cover letter and other job documents for an 
academic job search, so you should consult such resources. Additionally, you could ask for 
feedback from your professor on your job documents. If your professor has experience serving 
on a search committee, they would have some insights into preparing robust job applications. 


Resume or Curriculum Vitae (CV) 


CV is the factual list of your accomplishments, experiences, and competencies, and capabilities, 
including your educational degrees, professional experiences, research and scholarly 
achievement, and service work you have done. Career counselors suggest tailoring each CV 
when applying for different jobs. Suppose you are applying for a tenure-track assistant professor 
job in a research one university. Your CV’s content should highlight more about your research 
accomplishment and relevant research experiences according to the job description. In this case, 
your research accomplishment usually comes before your teaching experiences in the CV. 


Other knowledge and skills you could highlight in your CV could be related to research tools and 
applications. For example, being proficient in using the SPSS program to conduct quantitative 
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analysis, experienced in using NVivo to conduct qualitative data analysis, skillful in visualizing 
both quantitative and qualitative data, and proficient in analyzing video data are highly 
appreciated research skills. Finding a way to present these essential competencies you have in 
your CV is vital too. Your CV should reflect what you have done and demonstrate your 
commitment to and passion for the field of professional service. Like your educational 
background, research accomplishment, and teaching experiences, your professional assistance in 
your CV represents who you are and what you have done as a professional. 


To be successful at a tenure-track job search, you would want to have at least three or more 
articles published in peer-reviewed journals. Although having a first-authored paper would make 
your profile significantly stronger, co-authored publications also count, primarily if published in 
peer-reviewed journals. The CV must be factual and does not have many places to "tailor" it to 
match each job description. You can still tailor your CV to be research-focused by having the list 
of publications and grants at the beginning of the CV or make it more teaching-focused by listing 
your teaching experiences first. 


Letter of Reference 


Finally, reference letters also should demonstrate the match between your qualifications and job 
descriptions. Although your references decide the content of your recommendation letter, you 
can communicate with them individually about the nature of the jobs you are applying for and 
what kind of experiences you wish them to emphasize in their recommendation letters. It is also 
advisable to provide your references with your cover letter or letter of intent for the job 
application and your updated CV, both of which would provide your contacts with more focused 
information about the job you are applying for. 


Interview Preparation: Phone, Zoom, and One-Way Video Recording 


Once you successfully pass the search committee members’ document review, you could move 
to the next step --- an interview. Usually, there are two interview phases: the initial and the 
campus interviews. Before the pandemic, search committee members interview candidates by 
phone. Then, with new technologies, they started interviewing using web conferencing 
applications such as Zoom, Google Meet, or Webex. The same rule of showing your match to the 
job description persists in your interviews. Additionally, due to the stressful nature of job 
interviews, your preparation will be critical for achieving positive results. 


The first round of interviews usually happens remotely, either by a voice-only traditional phone 
call or by a video call using Zoom, Skype, and so on. In either case, you want to prepare an 
environment without distraction. You want to be in a quiet place so that the search committee 
will hear your voice, and you can listen to their voices. 


Initial Interview 


Format. The search committee chair facilitates the interview meeting with each member, asking 
at least a specific question. Sometimes, the search committee will ask permission to record the 
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interview session to share with non-attending members. In the end, you are allowed to ask 
questions about the search. 


However, do not be surprised if the search committee asks you to video record yourself 
answering interview questions. Not many colleges or universities are conducting this one-way 
online interview, but some places do. Your computer camera is activated to record your answers 
to specific questions given a time limit without the search committee members. You have an 
opportunity to redo your recorded response once or twice if not satisfied. 


Preparation. If you are going for a phone interview, you will face another type of difficulty. It 
would be best if you talked to multiple people on the phone. Since, in most cases, you don't 
know the voices of everyone in the search committee, you would have a hard time identifying 
who is speaking. If you can learn the names of the search committee before the interview, find 
their information online and print out their photos. Seeing their pictures while talking on the 
phone could make you feel less stressed in this awkward setting of talking to multiple people at a 
time. 


In the case of video interviews, be sure that you have a secure and robust internet connection. 
Internet interruption will result in a stressful interview and will not help your success. Make sure 
to dress business casual to show your respect as well as commonsense. If you are not sure of 
what "business casual" means, ask for advice from your professors and store staff. Check how 
the environment would look through the camera, too. The background must be clean so that you 
would look professional. The light probably does not want to be so dark, so you do not look 
creepy, either! 


Campus Interview 


Format. At the initial interview, the search committee will use the output to decide whether to 
invite you or not as a candidate for a campus or onsite interview. While waiting for an invitation, 
it is a good practice to send a thank-you note via email to the search committee chair for the 
opportunity to interview. 


Once you pass the phone or the video interviews, the next step would be an invitation for the on- 
campus interview. Most candidates cannot come this far, so you must be very proud of yourself 

if you get to this final stage of the job hunt. However, you should also be ready that this last step 
would be the most stressful, tiring, and scary part of the job hunting journey. 


The on-campus interview would consist of interviews with the search committee, the department 
chair, and the Dean. Sometimes students in the department will interview you as well. Hopefully, 
by now, you already know about the Search Committee members. Still, you will also need to do 
your homework to learn about the Department Chair and Dean and the missions, agendas, and 
initiatives in the Department, School, and the University. 


Preparation. The on-campus interview could be a stressful experience at the get-go. But 


interviews are not the only tasks during the campus visit. If you happen to receive an invitation 
to interview in another state, the campus visit starts as soon as you arrive at the nearby airport. 
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Typically, the search committee chair or member will come to the airport to pick you up and 
offer you a ride to the campus, restaurant, and hotel. Yes, you would most likely have a two-day 
visit. You will fly in at the end of the first day and interview and fly out the following day. 


On-campus interviews are essential because everyone evaluates or judges if you are the right fit 
for the position and the department, college, and university. The search committee chair or 
members gather as much information about you at every opportunity, such as airport pick- 
up/drop-off, dinner, or campus tour. You can only relax when you are in the hotel by yourself or 
in the restroom! Many consider the campus visit as the most tiring and stressful step in the job 
search. 


Presentation Preparation 


Additionally, in the cases of research-focused universities, you would be asked to provide a Job 
Talk where you present your research in front of the Search Committee and other faculty 
members in the department. The Job Talk looks very similar to your dissertation defense if this 
helps you visualize how it looks. 


Finally, in most cases, except for some research-focus universities, you will have another 
challenging task to complete, i.e., teaching demonstration. You will provide a lesson, typically 
about 45 minutes, in front of the search committee and other faculty members. Sometimes, the 
teaching demonstration happens before a group of students in the department. All groups 
evaluate your potential as a teacher in the program. It would be best if you showed your ability to 
teach content and have effective interactions with students. You would feel very awkward and 
nervous to teach in front of the people judging you, but a teaching demonstration is an excellent 
opportunity to showcase your teaching skills. In most universities, successfully teaching students 
is critical for professors, so you want to prepare well to impress professors and students in the 
teaching demonstration. 


Since the campus visit is so busy, stressful, and tiring, make sure to sleep well before the campus 
visit. You have a lot to prepare, but you will not have a successful campus visit if you sacrifice 
your sleep. Since teaching demonstrations and Job Talks happen at many universities, it is a 
great idea to prepare for them even before getting invited to a campus visit. In this way, you 
would feel less stressed and panicked even if asked to visit a campus in a week! 


Other Things to Consider 


While getting a sponsorship is critical if you are an international student, you need to recognize 
that fewer and fewer universities sponsor HIB and Green Cards due to the declining economy. If 
the job description says that the university will not provide the sponsorship while you need it, do 
not waste your time applying for the position. They cannot support you no matter how much you 
need it. Sometimes you figure out that the university does not provide sponsorship at the 
interview. Then, please take it as a great interview practice opportunity. Often, you don't know if 
they give the support until you go to the final round. You might feel tempted to ask if they 
provide the sponsorship, but it is usually best not to ask about "what they provide you" unless 
you get the job offer. Too many people negotiate before even getting a job, which gives a wrong 
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impression. Once the Search Committee is attracted to you enough to provide you with the job 
offer, that is when you get the privilege to negotiate for sponsorship and other requests. 


You can also refer to books on tenure-track job hunting, such as "The Professor is in: The 
essential guide to turning your Ph.D. into a job" by Dr. Karen Kelsky. Her book and others 
provide details about how to write a robust application and prepare for the interviews. 


Conclusion 


Many believe that job search ends when one gets a job. The authors who have been working for 
years say that the job search does not end at all. At least for the two of the authors, they have 
changed jobs three or four times. Career counselors state that most individuals change jobs four 
or five times during their career, if not a lifetime. 


As the authors look back to their own job search experiences, they suggest that it is vital to take 
note of successful strategies one did or learned from others. They also mentioned that it is also 
good to identify what did not work well and not use it again. Some noted that one needs to pay 
detailed attention in putting together the job application packet. Every hiring organization, 
academia or not, has its preference on a good CV, letter of application and references, statement 
of teaching, and research philosophy. Finally, keeping track of where one applies is essential. 
The worst thing that can happen to an application is sending a letter to an institution with the 
name. Much worse is a letter that speaks of qualifications missing in the job advertisement. 


The authors believe that searching for a job can be a full-time endeavor. The reflections and 
advice shared in the paper would be a good guide for planning to find the right job or a new 
position in the future. 
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Visual Literacy and COVID-19: 


Online Representations Connecting Learning and Impacting Teaching 


Danilo M. Baylen 
University of West Georgia 


Allyson Wilcox 
University of West Georgia 


Introduction 


Since March 2020, everyone’s worldview and experiences have changed due to the COVID-19 
pandemic. An already visual-rich online environment has been inundated by images, from charts 
to photographs, depicting changes in people’s lives due to a prolonged bout with an invisible 
enemy. Being cooped at home with nowhere to go and teaching remotely, the researchers 
ventured into studying what their students considered and shared in their classes online as part of 
the visual journaling activity. The paper explores how the changing online visual landscape as a 
phenomenon connects to learning and teaching contexts. 


Activity 


Starting with Summer 2020, the researchers collected artifacts submitted by students in a visual 
and media literacy course. The artifacts allowed the students to complete a journaling assignment 
identified as a visual-based reflection. For the task, the professor asked students to post a visual 
aligned to a prompt. They also wrote an accompanying narrative as a response to a prompt. 
Students post their visual and written responses in a designated online discussion "space." Most 
of the visuals came from websites, online news, or social media postings. 


Given that the students study visual and media literacy as a discipline, the prompts asked 
students to provide a brief narrative on what they see in alignment with course-related content. 
The writing entries focus on students’ thinking of the relationship between the shared image and 
visual or media literacy concepts, such as visual or design elements, principles, compositions, 
and angles of a camera shot). 


Element 
Visual design elements refer to the basic units of visual communication (Hagen & Golombisky, 


2013). Examples of visual or design elements include line, color, space, shape/form, size/scale, 
texture, and value. 
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Principle 


Visual or design principles facilitate the creation of an aesthetic appeal. The principles guide the 
work and interact with each other to maximize the user experience. Though there are no definite 
principles, researchers and practitioners identified some as focal point/emphasis, rhythm/pattern, 
balance, movement, contrast, repetition, alignment, proximity, and unity (Hagen & Golombisky, 
2013). 


Composition 


Researchers and practitioners identified visual or design composition style as "the arrangement 
of elements within a design" (Brown, Bussert, Hattwig, & Medaille, 2016, p.77). A typical 
composition technique is the rule of thirds. Other composition styles include leading lines, 
diagonal lines, framing, figure vs. ground, fill the frame, dominant eyes, and symmetry. 


Camera Angle 


The camera angle refers to how one composes a shot given the location of a camera about the 
subject. Researchers and practitioners classified the camera shot angle as eye-level (front or back 
of the object), high angle (top), low angle (bottom), or slanted (right or left). Using different 
camera angles to take a shot can provide different experiences for the viewer that may elicit an 
emotion. 


Visual Sharing Prompts 


To identify an image for the activity, students need an image that communicates or aligns with 
the prompts. Six descriptive statements provided the students with a focus for their visual sharing 
activity: 


1. An image of an opening event at your school (if you are not in a school, then your 
children’s school). 

2. An image of the management of the pandemic in your community. Provide a narrative 
reflection demonstrating the pandemic impacts on you or your family. 

3. Acartoon from online websites or social media that serves as commentary to the current 
political situation in the United States. 

4. An image that serves as a commentary on the current health situation in the United States. 

5. An image of children or young adults managing the daily challenges of living ina 


pandemic. 
6. An image that portrays positivity given the current situation (health, political, social, 
economics). 
Inquiry 


For the initial study of the visual artifacts, the researchers chose to collect and analyze the 
submissions from the first two prompts. Since the students study visual and media literacy, the 
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researchers wanted to know if image selection for sharing followed a pattern or theme. The 
researchers decided to focus on the following questions as part of the inquiry: 


1. What visual or design elements or principles are the students connecting within their 
journal entries on each topic? 

2. What strategies on visual or design composition styles and camera shot angles that the 
students connect within their journal entries on each topic? 

3. How did the students represent the impact of COVID-19 on teaching? 


Visual Coding and Analysis 


The researchers performed a "close" review of each image using the four visual or media literacy 
concepts specifically for pictures or photographs. The researchers collected 48 images for 
prompt #1 and 42 for #2. Some students submitted a photo for only one prompt. The researchers 
decided not to include them in the study. Also, the researchers did not include images with 
embedded texts or cartoons. The final image count for the study numbered 26 for each prompt. 


Once the researchers determined the images for analysis in the study, they coded them for the 
presence of a visual or design element and principle. Also, they identified the primary visual or 
design composition style and angle of the camera shot. After coding all the images, the 
researcher identified high frequencies in each category to answer the research questions. 


Findings 


Several patterns and themes emerged after analyzing the codes generated by the review of 
images in the four categories (i.e., elements, principles, compositions, angles). Table 1 identified 
those with the highest frequencies in each visual design category. 


Table 1 
Prompt Prompt Total 
#1 #2 
Element N=25 f INI=25 f N=50 f 
1. Space 23 0.92 17 0.60 40 0.80 
2. Color 14 0.56 23 0.92 aT 0.74 
3. Line i, 0.60 16 0.64 33 0.66 
Principle N= 25 ff N'=25 f N=50 if 
1. Repetition 16 0.64 13 0.52 29 0.58 
2. Focal Point/ Emphasis 14 0.56 14 0.56 28 0.56 
3. Contrast 21 0.84 6 0.20 es | 0.54 
4. Alignment 14 0.56 11 0.44 25 0.50 
Composition N=25 ve N=25 f N =50 f 
1. Patterns & Repetition 11 0.44 11 0.44 22 0.44 
2. Rule of Thirds 8 0.32 8 0.32 16 0.32 
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3. Leading Lines 9 0.36 4 0.16 13 0.26 
8 4 


4. Figure vs. Ground 0.32 0.16 12 0.24 

Camera Angle N=25 f N =25 ik N=50 f 

1. Right 8 0.32 Is 0.60 23 0.46 

2. Front 10 0.40 3 0.12 ie 0.26 
Elements 


The students submitted images that demonstrated elements of space, color, and line. These 
elements emerged as most common amongst student selections. The researchers coded the 
collected pictures based on these three elements and shape/form, size/scale, texture, and value. 
An example of an image from the study evidencing all three of these elements is displayed below 
(see Figure 2). 


Figure 2. An example of an image with space, color, and line. 


Principles 


The researchers coded the collected images using the category of visual design principles. After 
analyzing the coded images, the researchers identified repetition, focal point/emphasis, contrast, 
and alignment. See an example of a student-selected image demonstrating the visual design 
principle of repetition below (See Figure 3). 
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Figure 3. An example of the principle of repetition 


Composition Styles 


The researchers identified nine visual design composition styles to facilitate coding the collected 
images. After analyzing the codes on composition styles, the researchers found patterns and 
repetition, the rule of thirds, leading lines, and figure vs. ground, are highly evident. Below is an 
example of figure vs. ground composition style (see Figure 4). 


Figure 4. An example of figure vs. ground composition 


Camera Angles 


Finally, the researchers coded for angles of the camera shot among the collected images. The 
images selected and shared by students demonstrated camera shots taken using right or front 
angles. Below is an example of one of the images taken from a right angle (see Figure 5). 
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Figure 5. An example of a right angle shot of an image selected by a participant 


Connection to Teaching and Learning 


The pandemic impacted school activities at different levels. The pictures below (see Figure 6) 
demonstrate how COVID-19 changed how teachers and children worked together. The most 
common evidence of these changes is seen in these visual representations from using masks, 
teaching remotely, practicing social distance, checking temperatures, restructuring classroom 
space, and disinfecting hands. Masks became the new accessories in children's clothing for 
school. Teaching remotely became normal in conducting classes that challenged teaching 
practices for many teachers. If there is onsite teaching, then school staff rearrange desks, tables, 
and chairs to demonstrate social distancing and restructuring learning spaces. Finally, 
temperature checks and handwashing with disinfectants became the norms of everyday behaviors 
in some schools. 


Teaching Remotely Practicing Social Distance 


Checking Temperature Restructuring Space Disinfecting Hands 


Figure 6. Visual representation of COVID-19 impact on teaching and learning. 
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Challenges 


The researchers based the study on the output generated by a course activity. The activity asked 
students to reflect on the visual or design concepts learned and demonstrated the content 
visually. 


The first challenge that the researchers encountered involved understanding the student 
engagement and performance with the activity. The research explored the type of images 
submitted, and the researchers analyzed the visual or design characteristics. However, given the 
study's exploratory nature, the researchers began to think that the images' exploration as a 
research study needs reconceptualization towards a scholarship on teaching and learning. 


Another challenge the researchers experienced was the volume of data (both images and texts) 
available for analysis. The project started with over 40 pictures to code and decided to reduce the 
number to a more manageable number of 25 images. The researchers reduced the number of 
images in two ways. Initially, the researchers eliminated duplicate pictures submitted by 
students. For example, three students presented the same photo of a crowded hallway for their 
opening day prompt. The researchers analyzed the image once and removed the duplicates. Then, 
the researchers eliminated any picture that was not a photograph or with embedded text. In this 
manner, the researchers did not include political cartoons, infographics, and other non- 
photographic images. 


The data collection also included narratives written about the image by the students. The 
researchers decided not to code these narratives but focused on the images as the primary data 
source. The researchers chose to analyze the descriptions at a future date. They prioritize 
studying the images to facilitate understanding the context of what made students select the 
photos, and how they connected the images to teaching and learning, and the impact of COVID 
on those activities. 


Lessons learned 


Three significant lessons emerged from the research experience. First, a coding framework for 
classifying and categorizing the data around visual literacy components is beneficial. Next, 
similarities and differences could emerge between and among the images. Finally, previous 
experience with visual literacy played a vital role in data analysis. 


First, the researchers learned that a coding framework greatly aided the ability to classify and 
categorize data. They developed a coding framework around visual literacy components such as 
the elements, principles, composition styles, and angles of camera shots. The coding framework 
greatly assisted the junior researcher's ability to identify and code the images. 


Second, the researchers identified similarities and differences among and between images. For 
example, using the category of camera angles, the researchers found that students submitted 
many pictures taken from a front or right angle. The results did not surprise the researchers, 
given their experiences in taking photographs since many individuals choose to shoot photos due 
to being right-handed. 
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Third, the researchers found that experience and exposure to visual literacy concepts make a 
difference when analyzing photographs. Each researcher coded 25 images, but the amount of 
time taken by each researcher differed drastically based on personal experiences. The researcher 
with expertise in visual literacy coded the collected data in a shorter time, while the second took 
more time due to less familiarity with the content knowledge. The finding spoke to the need for 
training coders. 


Future actions 


The next step for the researchers will be to continue analyzing the collected images as data. The 
students received six prompts to respond to over the semester. The researchers analyzed the data 
from the first two prompts for this paper. They need to analyze the responses to the remaining 
prompts to understand the emerging patterns and themes in alignment with the content learned 
by the students. 


Another action needs to focus on analyzing the narrative portions of the students’ visual journal 
entries. Since each student responded to all prompts with images and narratives, the researchers 
also needed to code and analyzed them. 


After analyzing the images based on the visual or design categories, the researchers found that 
the students need more training on developing visual literacy knowledge and skills. Also, the 
researchers realized the importance of providing a tutorial on how to code images for consistency 
when using multiple coders. 


Finally, the researchers would endeavor to connect the information and experience gleaned from 
this study to teaching media literacy. 
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Leveraging ADHD: One Instructional Technology Professor’s Journey 


Dr. Joanne Beriswill 
Mississippi State University 
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Descriptors: Attention Deficit Hyperactive Disorder, Performance Technology 


Abstract 

This session chronicles the journey of one instructional technology professor’s journey from 
diagnosis and surviving with ADHD to thriving by leveraging ADHD. The highlight will be 
techniques for leveraging distraction and focusing the mind. 


How Did I “Get” ADHD-Inattentive Type 

Like many ADHDers, I found out that I had Attention Deficit Hyperactivity Disorder 
(ADHD) as an adult in a round about way. While browsing in the self-help section of a 
bookstore, I came upon ADD-friendly ways to organize your life by Kolberg and Nadeau (2002). 
I always wanted to “add” more organization to my life, so I read further. I soon realized that the 
“ADD” in the title actually stood for Attention Deficit Disorder (ADD), and that the descriptions 
of people in the book with ADD sounded a lot like me. Just like them, I was easily distracted, 
forgetful, late, etc., so I ended up buying the book. The tips were very helpful, but I never really 
accepted the fact that, like those described, I might have ADD. 

Fast forward a few years, and I was struggling to write my dissertation. In desperation, I 
enlisted the assistance of the director of the Writing Center at the University of Toledo, Dr. Carol 
Nelson-Burns. She suggested that I spend some time writing at the table in her office so that she 
could observe my writing process. After our session, she said to me, “As I watched you, you 
were listening to music on your headphones, moving along to whatever you heard, looking out 
the window at what was going on outside, AND typing away! Multitasking seems to be what 
helps you focus...and THAT seems classic ADHD—have you ever been tested for that?” At first, 
I was in shock; then, I came around and realized that it could be true. She suggested that I get 
tested to be sure of the diagnosis. After a number of sessions with a psychologist who 
specialized in ADHD testing, it was official—I had ADHD-Inattentive Type. 


Looking Back 

When going through the ADHD testing process, I was asked if I had any ADHD 
characteristics in childhood. While I am the opposite of hyperactive, I did have other ADHD- 
Inattentive traits as long as I can remember, such as staring into space, inability to concentrate, 
difficulties reading long passages or works, and needing music to concentrate on mental tasks. 
Beginning when I was about five, my brothers and sisters nicknamed me “the absent-minded 
professor” because I had trouble remembering everyday things but could recite volumes of 
information about whatever I found intriguing. Much to my parents’ chagrin, I tried a plethora 
of hobbies, only to leave them abruptly when my interest waned. 


After the Diagnosis 


I would love to say that after the diagnosis, I changed drastically and became a fully- 
functional member of society. However, that was not the case. The Ritalin I took gave me 
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amazing focus, but it only lasted for around five hours, and then I would crash. Timing was key. 
I had a short window of productivity for the day. I managed to complete my dissertation, but I 
realized that this would not be a long-term solution. That began a 14-year journey to explore 
ways to minimize distractions and build focus-friendly techniques. Each ADHDer is unique. 
What follows are some of the techniques that work for me. 


Characteristics of ADHD 

Although every ADHDer is unique, there are some characteristics that most of them 
share: porous short-term memory, an inconsistent sense of time, inconsistency of attention 
(distractibility and hyperfocus), a strong sense of adventure, and an aversion toward boredom. 


Porous Short-term Memory 

For some reason, the connection from short-term to long-term memory can be porous 
with only some information making it through the transition. This makes it easy for me to forget 
details of conversations and meetings. I use several techniques for increasing what I remember. 
Periodically, during conversations, I will stop and recap what was said. At the end of 
conversations, I summarize the key points, especially my next tasks. When possible during 
conversations and meetings, I take notes, which I process upon returning to my office computer. 

Another time when my porous memory affects me is when transitioning from one 
location to another. For example, sometimes I have to park in different lots around my building. 
After years of searching for my car, I developed a little whiteboard where I mark my car location 
when entering the office. I also have a list of items to remember upon leaving (keys, phone, 
jacket, etc.). When I arrive home, I take a moment to put things for the office on a shelf on a 
bookcase near the door that I call my takeoff zone. 


Inconsistent Sense of Time 

It can be very difficult for ADHDers to sense the passage of time. I personally can not 
tell how much time has passed—what I think has been one hour, sometimes is just a few minutes 
and vice versa. It is very easy for me to get caught up in a 5-minute task and find I am late for an 
appointment. Therefore, I use the alarms and timers available on my cellphone to remind me of 
my appointments, meetings, classes, and other events throughout the day. I find that I have to 
change the sound I use every periodically or I will turn them off absent-mindedly and return to 
what I was previously doing. Habits can also be helpful. To keep my daily reminders clear, I 
have trained myself to look at my calendar and to-do list each time I switch tasks. 


Distractibility 

Many ADHDers have sensory processing challenges. Therefore, they can be sensitive to 
distractions to their senses. I personally find that sounds, smells, touches, and sights can distract 
me while I work. For many decades, unfruitfully, I searched for distraction-free locations to 
work. However, about ten years ago, I found that I could create my own purposeful distractions 
to keep my senses occupied so I can concentrate better and focus on my work. I play music or 
movies (depending upon the task), burn a woodsy scented candle on my desk, wear clothing that 
is soft to the touch; and have minimal visual distractions in my work zone. These purposeful 
distractions keep my senses occupied so the rest of the environment does not distract me from 
my tasks. 
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Hyperfocus 

One of the greatest strengths of ADHD is the ability to hyperfocus. When we are 
motivated and free of distractions, ADHDers can enter a state of heightened focus where we can 
do tasks requiring intense concentration for an extended time. However, it is very difficult to 
maintain these periods consistently without following distractions like moving from searching 
for articles to transitioning to surfing the web. 

I have found that the extent of my productive hyperfocus is approximately 15 minutes. I 
use a modified Pomodoro Method with 15-minute tasks. I plan four to five pomodoro periods 
each day. Within each pomodoro, I have five 15-minute tasks. Following each 15-minute task is 
a 5-minute break, in which I do an activity like stretching, speed cleaning, or refreshing my 
drink. After completing all five of the pomodoro tasks, I take an extended 25-minute break to do 
something recreational, like taking a brisk walk, playing a computer game, eating lunch, 
watching TV, writing a personal letter, etc. Woven in the list of tasks for the pomodoros are time 
for meetings and appointments. I try to match my pomodoros with my workload with one 
pomodoro daily for research, teaching, and service with remaining pomodoros wherever needed 
most. 


Sense of Adventure 

Since one of the characteristics of ADHDers is their adventurousness, I try to be 
proactive and front-load adventure in my week so that I acknowledge and support my sense of 
adventure. For example, I try to drive home by a different way; I plan new experiences weekly; 
and am always saving for something special. Likewise, when I acquire some unexpected money, 
I have found Dave Ramsey’s philosophy of save some, spend some, blow some works best for 
my ADHD nature; otherwise, my sense of adventure will take over and I will spend much more 
than is wise. 


Aversion toward Boredom 

As an ADHDer, I also share an aversion toward boredom. When faced with a boring 
task, my instinct is to do something else. However, I can find that playing music or listening to a 
movie while I work can make monotonous tasks more palatable. I can also increase my 
adrenaline by setting a timer to induce me to speed up. When faced with a task that takes more 
time or effort, I resort to getting a person as a body double to help me stay on task. I always find 
it more fun to work with another person rather than alone. 


Give Myself a Little Grace 

I think the most important breakthrough I have made with my ADHD shortcomings is to 
give myself grace. I have learned to be gentle with myself and forgive the times I fall short by 
forgetfulness or the struggle to get things done. It is great to be able to step back and remember 
that I can choose to approach things with depression or a giggle, with pain or accepting that I am 
making progress, with guilt or grace. 


My Team 

Throughout my ADHD journey, I have had a team to support me. The biggest supporters 
have been my family. Many articles and books have been written on that topic, so I won’t dwell 
on that here. In addition, I have a psychiatrist, who has matched me with medication that helps 
me focus. I also have an outstanding ADHD coach who matches me with techniques and 
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strategies to get unstuck from inattention and anxiety, to set priorities, and to increase my focus. 
Since I live alone, I also have an extra helper (body double) who assists me with tasks around the 
house that have extreme distractibility and need concerted focus, like organizing the garage, 
doing spring cleaning, and my annual office organizational readjustment. My helper is an extra 
pair of hands, but she also keeps me on track by saying things like, ““What is your goal,” “You’re 
taking too long...hurry up,” “What are you thinking,” “Do you really need that” or “You are 
digressing.” 


Conclusions 

I love my creative, adventurous, challenging ADHD brain. It allows me to be the unique 
and bright person Iam. The key to being a professor with ADHD is working with my strengths 
to overcome the challenges with a large dose of grace. 
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Abstract 

The transition to college from high school is significant for at-risk students, especially as 
they tend to struggle with self-regulated learning skills when trying to adapt to the 
university environment. In an attempt to mitigate this challenge and assist students with 
this transition, many universities offer first-year experience courses. The purpose of this 
action-based research study was to evaluate the implementation of a gamified curriculum 
(EdApp) for at-risk students enrolled in a university first-year experience course. This 
seven-week action research study incorporated a gamified curriculum designed to 
increase self-regulatory learning skills (goal setting, strategic planning, task strategies, 
self-instruction help-seeking, and metacognitive monitoring) and motivation (choice, 
control, collaboration, challenge, constructing meaning, and consequences) for 10 
academically at-risk students enrolled in a first-year experience course. Qualitative and 
quantitative data was collected from the Learning and Study Strategy Inventory (LASSI) 
instrument, journal reflection assignments, a Final Self-Reflection Learning Quest, 
gamification elements, and learning management system (LMS) metrics. Findings from 
this study indicated that although only one subscale of the LASSI, Self Testing, was 
found to be statistically significant, correlations were found between various 
gamification elements and the subscales of Information Processing, Concentration, and 
Using Academic Resources. Additionally, as a result of various cycles of coding and the 
emergence of themes, findings suggested that students perceived the gamified 
curriculum as helping to improve their academic mindset, study habits, and motivation, 
all while making their learning easier. Implications for instructors considering the 
implementation of a gamified curriculum and future areas of research are offered. 

Keywords: EdApp, gamification curriculum, at risk students, first-year experience 

course, motivation, self-regulatory learning skills 
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The transition from high school to college is hard for the majority of students, but it 
is even more so for at-risk freshmen who tend to struggle with self-management skills and 
lower self-regulated learning skills when trying to adapt to the university environment (Sun 
et al., 2017; Tang & Wong, 2015; Vallerand & Blssonnette, 1992). The odds are stacked 
against them; over 40% of college students do not complete their degree in six years 
("IES", 2018), with 33% dropping out entirely (Shapiro et al., 2017). The freshman year is 
especially critical, with 28% of students dropping out before their sophomore year (Shapiro 
et al., 2017). In order to help mitigate these odds, many universities have developed first- 
year experience (FYE) courses to ease students’ transition to college life (Connolly et al., 
2016). Ideally, the FYE course helps new students adjust to the university by developing a 
better understanding of the learning process to acquire essential academic success. Students 
learn to adapt and apply appropriate academic strategies to their classes and learning 
experiences, effectively managing their time and priorities (Young, 2019). The research of 
Dembo and Seli (2016) supports that self-regulated learning skills can be a predictor of 
academic success. When self-regulated learners are engaged — they adapt their thoughts, 
their feelings, and their actions to impact their learning and academic success. Additionally, 
both motivational and cognitive interventions are needed to effectively increase self- 
regulated learning (Pintrich, 2004; Zimmerman, 2013). 

A gamified curriculum has the potential to be effective in a FYE course to address 
the issues of at-risk student motivation and self-regulated learning. Specifically, the use of 
game-based thinking, mechanics, aesthetics, and motivational design strategies has been 
successfully incorporated into curriculum design in order to promote learning (Fazamin et 
al., 2015; Kapp, 2012; Kim & Lee, 2015; Su & Cheng, 2015). One reason for this is that 
gamification has the potential to externally motivate students (Kumar & Khurana, 2012; 
Nah et al., 2014, Su & Cheng, 2015) which is key to its effectiveness (Burke, 2014; Sailer 
et al., 2017). The feedback associated with gamification can also empower students as self- 
regulated learners (Nicol & MacFarlane-Dick, 2006). Feedback strategies in gamification, 
such as rewards and incentives, can be effective at helping students set goals and reflect on 
successful learning methods (Dichev et al., 2018). Therefore, gamification may be a novel 
way to address both self-regulated learning and motivation for at-risk freshmen as part of a 
FYE course. With the promise of improved self-regulated learning and motivation, a 
gamified FYE course could very well be a suitable launching point for a student’s 
successful academic career. 

Purpose Statement 

The purpose of this action research study was to evaluate the implementation of a 
gamified curriculum for at-risk students enrolled in a FYE course at a 4-year university. 
The first research question in this study explored how, and in what ways, the 
implementation of a gamified curriculum impacted the self-regulated learning skills of at- 
risk students. The second question explored how the implementation of a gamified 
curriculum impacted the motivation of at-risk students. And the third question examined 
the perceptions that at-risk students might have about the gamified curriculum on the 
quality of their learning experience. 

Theoretical Framework 

There were three theoretical frameworks that underpinned use of a gamified 
curriculum for this study. First, consistent with behavioral approaches which embody 
Skinner’s theory of operant conditioning (Skinner, 1938), reinforcement schedules are 
used within a gamified curriculum by providing rewards, badges, and points at varying 
intervals order to maintain learners’ interest (Kapp, 2012) by providing positive 
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reinforcement (Woolfolk, 1998). Second, Ryan and Deci’s (2000) Self-Determination 
Theory suggests that learners become more self-determined and motivated when three 
basic needs are met: Autonomy, Competence, and Relatedness. When assessed in this 
regard, gamification has been found to increase motivation by emphasizing positive 
learning habits, or fostering task-meaningfulness through chunked goals as well as 
immediate and positive feedback (Harrold, 2015; Sailer et al., 2017). Third, Turner and 
Paris’s (1995) Six C’s of Motivation theory was used as it emphasizes Choice, Control, 
Collaboration, Challenge, Constructing Meaning, and Consequences, all of which are 
inherent in effective gamification practices. 

Methodology 
Participants and Setting 

The setting of this study was a FYE course at a regional campus in the South 
Eastern U.S. with an enrollment of about 2,000 students. The 16-week FYE section 
taught in Spring 2020 had 17 students. There were two inclusion criteria for the study— 
one was that the student had attended the university for at least one semester and that 
their GPA was less than 2.99, which is the institutional average. The other criterion was 
based on how, according to the provost, the university unofficially defines at-risk— 
students also could have failed a course to be eligible. After applying these criteria, 12 
students were eligible to participate. Secondary to COVID-19 restrictions forcing the 
campus to shut down, two of the students did not login or finish the course. Thus, the 
purposeful sample population size of my study that met the inclusion criteria when data 
was being collected was 10 students. Sixty-percent of the participants were male and 
forty-percent were female. 

Initially, the course met twice a week, in-person, for 75 minutes. The course was 
held in a classroom enabling each student access to a desktop computer where they could 
access the course content and all associated activities on the Blackboard Learn learning 
management system (LMS) during class meeting periods. After week 6, the COVID 
pandemic restrictions forced learning to be in the fully online environment. 

Research Design 

An action research approach supported trying out a novel gamified curriculum to help 
solve the problem of FYE students lacking self-regulated learning skills. Adjustments could be 
made through spiral of continuality, of implementing, evaluating, and revising the curriculum 
(Dick, 2002; Mertler, 2017). Use of a mixed-method design, with convergent strategies, 
offered the rigor of quantitative data analysis, coupled with the understanding gleaned from 
the qualitative data analysis. Triangulation was employed to ensure the findings were 
consistent regardless of the data collected or method utilized. 

Innovation 

The gamified curriculum was intentionally designed to help students acquire self- 
regulated learning skills through goal setting, strategic planning, task-strategies, meta- 
cognitive monitoring, help-seeking, and self-instruction. It was also designed to increase 
motivation, based on Turner and Paris’ (1995) 6C’s of motivation (see Table 1). 

The gamification elements designed in Blackboard consisted of worlds, quests, 
badges, currency, and a progress board. During week 4 of the semester, it became 
apparent students were not engaged in gamification as part of the Blackboard LMS as 


Table 1 
Self-Regulated Learning and Motivation Elements Aligned to Gamification Strategies 
Using EdApp 
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Self-Regulated Learning Elements 
* Goal Setting 


* Strategic Planning 
* Task Strategies 
¢ Self-Instruction 


* Help-Seeking 


¢ Meta-Cognitive Monitoring 


* Goal Setting 
* Strategic Planning 
* Task-Strategies 


* Goal Setting 
* Strategic Planning 
* Task Strategies 


Motivation Elements 


Choice 


Control 


Collaboration 


Constructing meaning 


Consequences 


Challenge 
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Gamification Elements and 
Strategies 

* Quests 

* Leveling-Up 

° XP 

* Currency (stars) 

¢ Achievements/Badges 

* Leaderboard 


* Gameplay 

* Quests 

* Leveling-Up 

* XP 

* Currency (stars) 
* Leaderboard 


¢ EdApp Customer Support 
Function 

* Group quests were not 
possible after everyone left 
campus due to COVID-19 


* Quests 

* Leveling-Up 

¢ Achievements/Badges 
«XP 

* Currency (stars) 

¢ Leaderboard 


¢ Achievements/Badges 
¢ Leaderboard 

«XP 

* Leveling-Up 

* Currency (stars) 


* Objectives 

* Quests 

* Leveling-Up 

¢ Achievements/Badges 
¢ Leaderboard 

«XP 

* Currency (stars) 


evidenced by their expressed apathy and dissatisfaction, lack of checking their achievements, 
and paucity of enthusiasm during class discussions. Seeking a solution, some computer 
science students conducted a focus group with the FYE students to see what they desired in a 
gamification platform. The results indicated the majority of students preferred competitive 
games, convenient social media apps, and a strong desire for a visually attractive app. The 
majority said points would serve as a motivating factor and that they would like a 
leaderboard highlighting those students in the top running. As well, they would find extra- 
credit opportunities worthwhile to them. The recommendation offered was EdApp, a mobile 
LMS app with gamification capabilities of holding all the course content, and students could 
access the platform on their phones like any other app. Over the next couple of weeks when 
school was postponed for Spring Break and the COVID-19 transitioned to fully online course 
delivery, all of the class content (quests) were created in EdApp. This resulted in a seven- 
week innovation implementation. Fortunately, due to the ease of the platform, very little 
explanation was needed and students dove right in. 

When students clicked on a quest, they experienced a journey with an opening 
objective followed by a mix of multiple-choice, free-response, Likert-Scale and true/false 
questions, words of encouragement, as well as various games. Slides were not static and there 
was always an element of interaction, whether it be pressing a button, swiping, dragging and 
dropping in a word, scrolling to select a number, circling the correct answer, drawing a line to 
associated items, expanding a bulleted list, or flipping a card for more detail, etc. 
Supplementary visuals and videos were also included so information could be expanded 
upon, or examined in more detail by students. Each main lesson quest concluded with a 
Jeopardy-style game in which students could win experience points (XP) and stars. Ending 
quest slides either offered encouragement, reminded students of the star bar opportunity, or 
an extra-credit mini-quest of the same name comprised of game opportunities. At the end of 
each extra-credit quest, students were encouraged to go to the star bar to spend their stars for 
a chance to win an Amazon gift certificate. In addition to the quests in EdApp, students were 
able to access the class leaderboard (populated by achievement points earned for each quest 
activity), the star bar (students earned star currency when answering reinforcement questions 
or engaging with the game slides in EdApp), their performance metrics (stars earned, lesson 
completion status, and badges earned), and the Brain Boost quiz function via the side menu in 
the app. See Figures | through Figures 4 for the design of EdApp used for this study. 


Figure 1 Figure 2 
Structure in Ed App Answering Options 
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Figure 3 Figure 4 
Badging Opportunities Game Currency: Star Bar 


BS SRR Be SSB eee ogD! 


Data Collection 

There were four data collection sources in this study: 1. Gamification Element 
Metrics; 2. Final Self-Reflection Learning Quest—which was comprised of free-response, 
multiple choice, and Likert-scale questions; 3. Reflection Journal Assignments, which 
consisted of two to three free response prompts; and 4. the Learning and Study Strategy 
Inventory (LASSI, Weinstein et al., 1987) which was administered to students at the 
beginning and end of the course. This survey measured the ten subscales of Anxiety, 
Attitude, Time Management, Test Strategies, Self Testing, Using Academic Resources, 
Concentration, Information Processing, Motivation, and Selecting Main Ideas. Each subscale 
was measured by the extent to which students agreed or disagreed, using a Likert-scale, with 
each of six reflection statements. To ensure rigor and trustworthiness, triangulation, member 
checking, thick, rich descriptions, peer debriefing, and an audit trail was conducted. 

Results 

Between the various LASSI subscale scores, the multiple-choice and Likert-scale 
questions used in the Final Self-Reflection Learning Quest, as well as the numbers of 
gamification elements and metrics, this study yielded sizeable quantitative data. The LASSI pre 
and posttest scores were analyzed using paired t-tests. The Bonferroni adjustment was used to 
help guard against the possibility of false positive findings since there were multiple measures to 
address a single research question. The findings revealed two subscales decreased, one remained 
the same, and seven increased—with only three of those being significant. After applying the 
Bonferroni adjustment, only the Self-Testing subscale showed statistical significance (p<.001) 
meaning participants' self-regulation regarding learning and study strategy skills improved after 
experiencing the gamified curriculum innovation. Descriptive and frequency statistics were used 
on the numbers of gamification elements. Without any context, it was difficult to ascertain 
whether or not these elements had intrinsic value. Therefore, a series of correlation analyses was 
performed, with Pearson’s r as the correlation coefficient, in order to identify statistically 
significant relationships between the LASSI subscale scores and the gamification element scale 
values. The findings indicated that the Information Processing posttest score strongly correlated 
with experience points (or “XP”, r=.90, p=<.001), as well as other measured variables, such as 
the number of stars earned (where in the context of EdApp, “stars” are a form of “currency”, 
r=.89, p=<.001). Information Processing also strongly correlated with the chance of winning a 
gift card motivating the student to do better in class (r=.85, p<.0/). The gift card ranged from $5 
to $20 during the course increasing the student’s desire to win one (7=.82, p=.002). The 
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Concentration posttest score strongly correlated with badges earned (r=.8/, p<.0/), and Using 
Academic Resources strongly correlated with the number of stars earned (r=. 80, p<.0/). 

For the qualitative data, the process of inductive analysis began with initial rounds of 
coding that included Structural, InVivo, Process, and Descriptive coding (Saldafia, 2016). During 
the second cycle of coding, Pattern coding resulted in the emergence of 10 categories and 
subsequently, three themes. The first theme was students perceived their academic mindset and 
study habits to have improved which denotes an improvement in study skills as well as growth in 
terms of one’s academic mindset. It is distinguishable from the other two themes in that it was 
tied to students taking the academic skills they have learned and applying them to their other 
classes. The second theme, the gamified curriculum served to motivate students was 
distinguishable from the other two themes, in that it involved more “will” to learn, as opposed to 
skill and ease of learning. The third theme, students perceived the gamified curriculum made 
their learning easier effectively addressed the impact that the content’s delivery vessel had on 
students’ experience. 

Although the LASSI Motivation subscale was not found to be statistically significant 
(p=.08), it should be noted that the LASSI measured motivation across the board as it pertained 
to the students’ total academic experience, which included other classes. It does not mean the 
gamified environment did not improve motivation in this class specifically, as evidenced by the 
many positive qualitative responses that the students provided (i.e., “The class has motivated me 
to become less lazy and get things done.”, “This class motivated myself to manage my time 
when coming to studying for a test and having to finish something with getting enough time to 
sleep for class the next day”, “I enjoy it. I like to compete and the games engaged me more in the 
classroom.”’). Virtual currency was also a big motivator for some students, especially those who 
participated in a chance to win a gift card in the star bar (M=4.5/6.0, SD = 0.90). As shared by 
one student, “The stars motivated me to finish my quests. The stars were my biggest motivation 
because I wanted to play in the star bar.” Also, because the students found relevance in the 
content, they were more motivated to use what they learned outside of class. Three students 
expressed this in their final self-reflection learning quest responses, ““When we went over the 
studying unit and how much you really should study, I applied that studying to my 
macroeconomics exam.”, “This class helps you with your other classes and you can apply your 
knowledge.”, and “I bettered my study habits in my history class.” 

Overall, the students perceived the experience on EdApp to be quite appealing—it was 
very well received, especially in terms of its clean, organized structure free of distractions, its 
perceived stress-free environment, its ease of use, and the opportunity to have all content right at 
one’s fingertips on a mobile device. As stated by a couple students’ final self-reflection learning 
quest responses, “This app sort of felt like a break and I enjoyed learning the information. My 
other classes are like stress and work. This was more of a weight off of my chest.”, “I like the 
layout of the app, everything is well organized.”, and “You can use it on the go with your phone 
and it was convenient if I was someplace else.” 

Discussion 

It is essential to look at findings within current research regarding gamification in an 
educational context. To address the research questions, data were merged and analyzed through a 
mindset of self-regulated learning, motivation, and perceptions about gamified curricula. 

The first research question this study explored was how, and in what ways, the 
implementation of a gamified curriculum impacted the self-regulated learning skills of at- 
risk students. The gamified elements of the course, as delivered through the EdApp 
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platform, gave students an enjoyable venue in which to improve upon their own self- 
regulated learning as part of the course. This was not surprising, as when college students 
are given autonomy in gamified environments, they tend to have stronger self-regulated 
learning skills than students in traditional, controlled settings (Lambert, 2017). What made 
this gamified experience unique was that the students were actually learning how to 
improve their study habits through the curriculum content in addition to engaging with the 
gamified elements, thus providing them with a double dose of opportunity to improve in the 
area of self-regulated learning. As shared by one student, “The test-taking and study 
strategies were the most useful for me because test-taking and studying is usually what I 
have the most difficulty in.” The opportunity for personal reflection allowed students to 
apply what they learned to their own lives and find meaning, showing how a gamification 
platform could be considered an ideal way to teach self-regulated learning skills such as 
goal setting and persistence through a growth mindset, especially as games exemplify these 
characteristics (Devedzic & Jovanovic, 2015; Educause, 2014; Gibson et al., 2015; Sailer et 
al., 2013; Tang & Kay, 2014). This was supported by a student’s comment, “the course 
really helped me self-evaluate and better prepare myself for the future. I have not had a 
course that was so about myself and the critique and critical thinking of one’s self- 
evaluation on schoolwork and everyday things.” Students who are engaged and actively 
generating meaning while adapting their thoughts, feelings, and actions as necessary to 
affect their learning and motivation are considered to be self-regulated learners (Boekaerts 
& Corno, 2005). This can be accomplished with both video and written responses within 
the gamification platform as a form of metacognitive scaffolding to improve self-regulated 
learning (Tang & Wong, 2015). 

The second research question explored how the implementation of a gamified 
curriculum could impact the motivation of at-risk students. Gamification has been found in 
the existing literature to yield increased motivational results (Harrold, 2015; Ling, 2018; 
Pilkington, 2018; Sailer et al., 2017). Quantitatively, this was not the case for the students in 
this study as the LASSI Motivation subscale did not indicate a statistically significant 
increase in score. However, qualitatively students did offer comments in their Final Self- 
Reflection Learning Quest about experiencing an increased sense of motivation through the 
use of the gamification curriculum. As evidenced by student responses on the final self- 
reflection learning quest, “After the class I’ve been really been motivated to do all of my 
assignments”, or “pushed you to better yourself because everyone can better themselves.” 
The integration of virtual currency through the form of being awarded stars in the EdApp 
curriculum was identified by students to be a source of motivation. As shared by one student, 
“The Amazon gift card thing was usually a challenge. I would end up spending all of my stars 
while trying to win.” Additionally, the amount of engagement in earning badges, stars, and 
XP, which in turn resulted in students’ completion of extra credit quests, are attributes 
directly associated with motivation (Yot-Dominguez & Marcelo, 2017). 

Motivation through the use of a gamification curriculum can also be seen through 
feelings of competence and relatedness (Sailer et al., 2017). In order for competence to be 
experienced by the learner, gamified activities should pose optimal challenges to the student 
(Kam & Umar, 2018). Challenges in gamified curriculums can predict student learning 
while increasing engagement (Hamari et al., 2016). In this study, students voiced their 
pleasure in terms of the challenges they conquered in the course; which supported the notion 
that the motivational appeal of games may be their ability to provide players with challenges 
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to master, thereby enabling feelings of greater competence (Mekler et al., 2017). Meeting 
students’ needs for relatedness can be fostered by creating shared goals (Sailer et al., 2017) 
and achievements (Sillaots, 2015). In this study, a collaboration did not result from peers in 
class working together but rather from the students’ collaboration having reached out to 
other students in the university tutoring center for help. As found in one student’s reflection 
journal assignment response, “I was not suffering in silence anymore as I started going to the 
tutoring center.” In support of this particular student’s response, her score on the LASSI 
Using Academic Resources subscale showed an increase of 150% from the pretest to the 
posttest. 

The third research question examined the perceptions that at-risk students might 
have about the gamified curriculum on the quality of their learning experience. Overall, this 
study’s students’ perceptions of their experience with the gamified curriculum using EdApp 
were very positive, and everyone shared that they liked the design of EdApp. In fact, they 
wanted more gamification, as found in a couple open-ended final self-reflection learning 
quest responses, “the whole course should be on the EdApp with the games”, “use EdApp 
and make the whole class online.”, “I wished we would have switched to EdApp sooner”, 
and “it [EdApp] should keep going for years to come and be implemented in education as a 
whole.” Although students’ Concentration and Anxiety LASSI subscale scores did not show 
statistically significant improvement in their overall academic experience at college, there 
was a strong correlation between Concentration and badges earned. Moreover, students 
identified the gamified curriculum as making the content less distracting. As evident by two 
student responses, “I never felt distracted from the course,” and “I felt less distracted in this 
course than any of my other courses.” Similar positive student perceptions of the 
gamification elements were also found in the research of O’Connor and Cardona (2019). 
Gamification has been suggested as a platform to reduce anxiety levels in students (Paniagua 
et al., 2019) by offering low-stakes learning environments and allowing an opportunity for 
failure, which appeals to first-year college students (O’Brien & Pitera, 2019). As seen in one 
student response, “This app sort of felt like a break and I enjoyed learning the information. 
My other classes are like stress and work. This was more of a weight off of my chest.” Other 
students as well perceived this class to be a stress reducing force in their lives. 

As observed between this study’s outcomes and that of Brom et al. (2019), students 
were indifferent to XP as a standalone gamification element, but they were more vocal in 
their attitudes regarding XP as displayed on the leaderboards. As one student expressed, 
““XPs in general were more like a bonus to me.” Additionally, students can learn to view 
failure as an opportunity instead of becoming overwhelmed and helpless (Lee & Hammer, 
2011). Students in this research aligned with the student perceptions of the research by 
Dicheva et al. (2019) as well as the research by Donovan et al. (2013) in that they tended to 
be more excited about the virtual star currency than badges, as shared by one student who 
felt “like a BOSS!!!” However, leaderboards and badges can also promote a sense of 
competence (Bai et al., 2020; Sailer et al., 2017), which was the case for one student who 
responded on the final self-reflection learning quest that she felt like she had “accomplished 
something” when earning a badge. 

Student responses in this study reflect those of Shroff et al. (2020) showing that they 
really liked EdApp as it met the characteristics of being transparent, fun to use, and 
aesthetically pleasing; having a comprehensible organization of course content; and being 
easily accessible at all times of the day. As a couple students expressed in their reflection 
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journal assignment responses when asked to compare the use of EdApp and Blackboard 
Learn, “It’s more interactive. It just felt cool. It was more than I was expecting.”, “EdApp is 
fun, colorful, and interesting,”, and “I love that games are part of the lesson and the stars 
remind me of coins in a game.” Many students also noted that EdApp was easy to use, “It is 
fast and smooth, making it really not annoying to work on.” and “I like the layout of the app, 
it is well organized.” However, it should be noted that students did encounter some hiccups 
with EdApp as well, “It made me restart the lesson a few times” and “‘it did glitch a little”. 
However, these glitches did not appear to temper their perception of the experience overall. 
Implications 

Gamification research should not be restricted to motivation, satisfaction, academic 
achievement, and engagement; it should also include the potential to promote teamwork and 
group cohesion (Bilgin & Gul, 2020). In a future implementation, it is recommended to add 
items to quests, perhaps adding extra-credit quests, to promote teamwork (Donovan et al., 
2013) while realizing a more socially interactive experience that can help users develop social 
competence (Tang et al., 2020) through cooperative and collective gamification approaches 
(Koivisto & Hamari, 2019). An emphasis on a more social gamification experience would 
likely help build the participant’s social status, resulting in better retention rates and skill 
acquisition (De-Marcos et al., 2016). Use of the video discussion feature in EdApp, asking 
students to comment on various questions and concepts via the video chat, could help 
accomplish this. 

An important element of incorporating gamification elements into academic courses is 
to provide students with a sense of control over how their learning takes place (Shroff et al., 
2020). Moreover, when college students are given autonomy and choices through 
gamification, they tend to have stronger self-regulated learning skills than students in 
traditional, controlled settings (Lambert, 2017). Incorporating more choices of quests and 
activities within them, supports the notion that learners should believe they have freedom as a 
result of their own decisions to choose tasks or challenges presented to them (Turner & Paris, 
1995). Clear goals can help structure the learning task and increase the learner’s feeling of 
competency and sense of autonomy (Brom et al., 2019). Learners who have a clear goal are 
more likely to complete a task than those who are simply told to do their best (Jung et al., 
2010). During this study, badges were awarded for completing quests; however, it could have 
been more effective to have them match specific objectives or goals of the course (Bai et al., 
2020). Incorporating badges into a leaderboard as a different means of social comparison than 
points (Bai et al., 2020) or markers (Hamari, 2017) may be something to consider, especially 
as leaderboards require participants to set their own goals, striving to place themselves at the 
top (Landers et al., 2017). 

Other FYE instructors should keep in mind that student centered learning with the 
teacher as a moderator instead of lecturer, is essential in gamified environments (Lengyel, 
2020). Traditional content can be refined with key salient points being emphasized on the 
gamified platform. Students can be savvy consumers that expect technical accessibility and 
adaptability of content at their fingertips, which can be addressed by the adoption of mobile 
apps to deliver gamified curriculum in a higher education setting. Effective gamification has 
the potential to help with retention, especially if delivered on a mobile app, offering the 
students constant accessibility (Pechenkina et al., 2017). It has been said that gamification is 
not about technology or a digital platform but rather the design and development of 
innovative instruction which incorporates game elements into activities (Zainuddin et al., 
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2020). However, the technology is not irrelevant. Use of the EdApp platform made the 
delivery of instruction easier and better in this study. This platform had preformatting that 
was applied to all text, which helped to make content look inviting and consistent. Instead of 
offering a busy PowerPoint, EdApp distilled that content down to salient points to be 
presented with ample white space and the capacity for scrolling and swiping as mandated by 
the necessary constraints of the EdApp platform. This ensured a clear and effective 
experience for the student users. In order to keep the FYE intimate, there needs to be plenty 
of opportunity for self-reflection and communication. Given that students had numerous 
opportunities to self-reflect, they were able to ascribe their own relevance to content and the 
experience. The EdApp gamified curriculum allowed students the freedom to fail and try 
again, which embodies the Growth Mindset (Dweck, 2007) at its core. As shared by one 
student, “The aspect that motivated me the most was the growth mindset quests.” In contrast 
to the assumption that gamification is more time-consuming for the instructor—especially 
with technical issues (Daubenfeld & Zenker, 2015) and the increased amount of grading 
required in order to keep up with rewards and achievements (Evans, 2016)—a key benefit of 
gamification is that once the content is loaded into a platform like EdApp, the professor will 
have sufficient time to reach out to students personally and to spend time thoughtfully 
responding to their reflections. It should be noted, however, that the time commitment in 
setting up this system on the backend is significant (Bratt, 2020). 

More gamification FYE studies are needed—especially those that are participant-focused 
and examine different types of learners. Studies need to be larger and longer, and they could 
examine personal versus private dashboards, especially in terms of goal-setting. Research should 
also be done to measure the effectiveness of different learning management systems and their 
potential to deliver a gamified curriculum. Implementing gamification platforms can change, for 
the better, how classes are taught and how the content is presented. It has been suggested that, 
similar to the swiftly changing field of gaming, the study and usage of gamification requires a 
constant review of research findings as it continues to evolve (Hulsey, 2015) both 
technologically and pedagogically (Banfield & Wilkerson, 2014; Barneva et al., 2017; Toyama, 
2015). 

Limitations 

The most unexpected limitation of this study was the COVID-19 pandemic, which sent 
students home, and moved the class entirely online. The pandemic affected students’ dispositions 
both in terms of motivation and turning in assignments in a timely manner. Other limitations 
include that of action research itself, which is not generalizable as well as the novelty effect 
(Clarke & Sugrue, 1988) that is common in gamification studies (Hamari et al., 2014; Hanus & 
Fox, 2015). And, of course, there are the challenges associated with self-reported data. The more 
widespread gamification LMS platforms become, and the more they compete for adoption, the 
more they are likely to continually refine and update their product, which will free the instructor 
to focus more on student relationships and keeping their content, not the platform technology, 
current. 
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Abstract 


An introductory pharmaceutics course was redesigned before the pandemic to integrate 
deep-learning active tasks. The primary objective of this research proposal was to analyze the 
design transfer for an online synchronous context and monitor its impact on students' 
performance, perceptions, and beliefs. We found that highly integrated active learning tasks 
effectively transferred to the online format. In addition, student performance, perceptions, and 
opinions were similar or slightly better when compared with the live form of the course. 


Motivation and Objectives of the Study 


The COVID-19 pandemic restrictions at the local university at the beginning of the Fall 
2020 semester allowed the instructors to decide whether the course will be offered in mixed- 
attendance format with both in-class and online groups or fully synchronous online. At the same 
time, when the instructor provided mixed attendance, students could decide if they would use the 
live or online option for attendance. For the courses that offered live attendance options, the 
state-wide distancing strategies were implemented, and masks were required. 

For this research study, the focus was an introductory pharmaceutics course 
(Pharmaceutics I) that was redesigned before the COVID-19 pandemic to integrate active tasks 
such as in-class instant feedback with a personal response system (clickers) as well as deep- 
learning strategies such as productive failure (Cernusca & Mallik, 2018) and concept mapping 
(Cernusca & Mallik, 2020). When faced, due to COVID-19 pandemic restriction, with the choice 
of a mixed-attendance or fully online option for the pharmaceutics course, the instructor worked 
with an instructional designer and decided to implement a fully synchronous online option to 
adequately transfer the benefits of the active learning strategies already built in the course 
design. 

The primary objective of this study was to describe the impact of the design constraints 
and affordances specific to the local COVID-19 pandemic context on the nature of the course 
implementation and analyze the impact of the design transfer to online format on students' 
performance, perceptions, and beliefs. 


Instructional Design Affordances and Constraints 


Design in general, and instructional design as one of its subsets, is a complex, ill- 
structured, and context-driven problem-solving process (e.g., Jonassen, 2011). Therefore, as a 
first step in designing or redesigning an instructional process, is the analysis of the design 
constraints and affordances specific to the target context of the instruction (Larson & Lockee, 
2014). 
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Design Constraints 


The target Pharmaceutics I course was built, at the time of this study, on a heavy 
integration of the active learning tasks into the lectures that ranged from basic active learning 
tasks (e.g., clicker questions) to more extensive strategies such as productive failure and concept 
mapping. Before the course started, because of the COVID-19 pandemic distancing 

requirements (Figure 1), the instructor 
Figure | had the option to use a mixed-attendance 
Typical COVID-19 Pandemic Classroom option and split the class into two groups: 
one group came to the class and the other 
attending remotely through Zoom. 
Besides the issues related to ensuring an 
equitable split of the students for the two 
attendance groups, several instructional 
constraints associated with selecting the 
mixed-attendance option were found. 

First, the classroom designated for 
the course was not fitted with audio and 
video technology to seamlessly integrate 
distance students in the live classroom. 
Consequently, a mixed-audience option 
would not allow integrating the live and 
remote students during the discussions 
associated with the hands-on activities in 
the course. Moreover, even for the 
students in the classroom, the 
requirements of safe distancing combined with the use of masks would have been hardly 
conducive to teamwork. Second, with a class size of 62 students, the instructor decided that it 
would be very hard, if not impossible, to fully and effectively integrate the live and the remote 
groups during the class time, especially without any previous training and a very short time to 
prepare for this type of classroom setting. 


Design Affordances 


On the other hand, when the existing course design and implementation were analyzed, 
several design affordances pointed toward the benefits of a fully online synchronous format of 
the course. 

First, the lecture materials used in the course were already in a digital format, PowerPoint 
slides. Therefore, these instructional materials were readily transferable to an online synchronous 
form using online conferencing platforms such as Blackboard Collaborate Ultra or Zoom, both 
available to the instructor at the time of this study. Second, and even more critical for the overall 
effectiveness of the instructional process, active learning tasks were either already using a virtual 
tool or strategy (e.g., virtual clicker questions for on-time feedback) or directly transferable to 
online synchronous format (e.g., breakout rooms for group activities). 
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For example, active learning tasks already integrated into the instructional process, such 
as think-pair-share and productive failure, used virtual clickers in the live format and required no 
accommodations to transfer them online (see Figure 2). 

Finally, the exams were set 
up in the live format of the course 


Figure 2 as open-book, and the only 
Virtual Clicker Question Used During Lectures restriction imposed to the students 
during the exam was to not 
POP Quiz communicate with peers. This 


exam-related requirement could be 


It is hydrolyzed by water to enalaprilate. What is the overall order of the reaction? easily enforced by requiring the 
wae students to use a second device that 


Enalapril is used as an oral medication to treat hypertension and congestive heart failure. 


we po will allow live monitoring of their 
—«f 7 ff e, working space in the online 
7 107 So conferencing environment used for 
mene" the course. 


the Keywords: 


= Matched: 


. Transfer of the Course to Online 
Synchronous Format 


tis 


The analysis of the design barriers and affordances indicated that the synchronous online 
format of the course would fully use the identified design affordances and minimize the potential 
negative impact of design constraints. Also, a pre-course short survey sent by the instructor to 
the course cohort provided support for the above decision with most of the students either having 
a strong preference for online format of the course or no preference for a specific course format 
and only less than a fourth of the students indicating a strong preference for the live form of the 
course. 

At the same time, the above analysis indicated that the instructor's preparedness for the 
online course administration was critical for the transition to the fully online format of the 
course. To ensure the effectiveness of the instructional process, the instructor set up a well- 
equipped office with a second monitor and HD webcam that allowed a smooth control of the 
course materials and a continuous monitoring of the chatroom. The online course was 
administered with Zoom® (https://zoom.us/) conferencing platform. All the online features 
required as part of the synchronous online course administration, from the chatroom to breakout 
rooms, and the use of the virtual clicker were pre-tested by the instructor with the help of an 
instructional designer and the teaching assistants assigned for this course. This process continued 
during the semester through short debriefing meetings between the instructor and the 
instructional designer and resulted in fine-tune adjustments to the online instructional process. 
Once the course started, the instructor kept his camera on and encouraged all students to keep 
their cameras on to get as close as possible to the live format of the course. On average, about 
90% of the students complied to this request from the instructor and showed their cameras 
during the course (Figure 3). 
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Figure 3 
Sample Lecture Zoom Wall-View 


During the online synchronous 
lectures, the instructor successfully 
integrated active learning strategies such 
as clicker questions and concept mapping 
to support the problem-solving process 
and productive failure to provide 
additional instructional support for 
difficult problems. All these strategies 
were built and tested as part of live 
activities prior to the COVID-19 
pandemic. 

For example, Figure 4 shows a 
sequence of steps associated with a 
practice problem for which the instructor 
used a combination of concept mapping 

; conceptual guiding strategy and clicker 
questions as support for the problem- solving: process associated with worked examples used in 
the lecture. 


Figure 4 
Integration of Concept Maps and Virtual Clickers During Online Synchronous Lectures 


Finally, for the team activities that ranged from the think-pair-share tasks associated with 
knowledge testing clicker questions to problem-solving, the instructor randomly split the class 
into small groups using the breakout room feature in Zoom. Then, when the problem-solving was 
the focus of the group activity, the instructor asked a representative from each group to type their 
answer in the chatroom and used the students' chatroom input to move to the next step in the 
instructional process. 

In addition, at the beginning of the semester, the instructor decided to monitor the overall 
pandemic situation for one month and decide at that time, along with his students, if a switch to 
hybrid attendance would be preferred. However, when students were asked about the potential 
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change in the course format, they selected the fully online synchronous option for the entire 
semester. 
Design Transfer Effectiveness: Research Methodology 


The instructor worked with an instructional designer both before and during the COVID 
19 pandemic to monitor the significant active learning instructional tasks implemented in this 
course to identify possible improvements in the course structure. 


Research Design 


We used a comparative exploratory quantitative research design for this study 
complemented with qualitative triangulation data from students’ course evaluations. The 
analyzed dependent variables were students' course self-efficacy, perceived course difficulty, 
perceived impact of productive failure, the perceived impact of concept mapping on students! 
learning and student performance outcomes for two assessments significant for the major active 
learning tasks implemented in this course. The independent variable was the cohort and had two 
levels: pre-pandemic face-to-face instruction and CODIV-19 fully online synchronous 
instruction. 


Data Collection 


Every semester the instructor administered a prior knowledge survey to test the 
homogeneity of the cohorts. In addition, an exit survey that included believes and perception 
constructs adapted from the literature was administered online using Qualtrics during the last two 
weeks of the course. The exit survey included a beliefs construct, self-efficacy, adapted from 
Bham et al. (2011) and three perception constructs, one related to perceived course difficulty 
adapted from Kappelman (1995) and two related to the perceived impact of productive failure 
and respectively concept maps, adapted from Grasman et al. (2013). All constructs were 
evaluated on a 1-low to 9-high scale, and most of them showed a very strong internal reliability 
above the 0.70 benchmarks suggested in the literature, as follows: self-efficacy, Cronbach's 
Alpha=0.91; perceived difficulty, Cronbach's Alpha=0.62; the perceived impact of concept maps 
and respectively productive failure both with Cronbach's Alpha=0.97. 

All students enrolled in the course were invited to participate in this study. Participation 
was voluntary, and there was no payment or grading reward for participating in this study. For 
the face-to-face group (Fall 2019) of the 75 students enrolled in the course, 74 (99%) completed 
the prior knowledge quiz and 56 (75%) completed the exit survey. For the online group (Fall 
2020), of the 62 students enrolled in the course, 60 (97%) completed the prior knowledge quiz, 
and 36 (58%) fully completed the exit survey. 


Data Analysis and Results 


Data was analyzed using SPSS v27®. An independent sample t-test indicated that the 
exam mean score for the online cohort (38.33%) was slightly higher than for the pre-pandemic 
cohort (38.12%). However, there was no statistically significant difference between the mean 
scores of the two cohorts (p = 0.93). This finding showed that the two cohorts were 
homogeneous at the entry point in the course. 
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To analyze if the transfer of the target course to the online format was successful in the 
first stage, we examined students’ perceptions of the major instructional strategies implemented 
in the course and the course as a whole. First, an independent sample t-test indicated that for the 
perceived impact of productive failure. At the same time, online students' mean score (6.61, on a 
scale of 1-low to 9-high) was lower than for the live cohort (6.95). However, the difference was 
not statistically significant (p = 0.36). Similarly, the mean score for the perceived impact of 
concept maps for the online cohort (6.59) was lower than for the live cohort (7.21), but the 
difference was not statistically significant (p = 0.11). Second, an independent-sample t-test 
indicated a statistically significant lower perception of online course difficulty, 4.94 compared to 
the pre-pandemic face-to-face format of the course, 5.96, t (90) = 3.55, p< 0.01. 

In a second stage, we analyzed two factors related to student performance, epistemic 
beliefs, typically seen as a proxy for learners' future performance, and respectively student 
performance on the two assessments, second and third exams, that measured the impact of 
productive failure and concept mapping strategies. First, an independent-sample t-test showed 
that the mean scores for the online cohort (7.06, on a scale from 1-low to 9-high) were 
practically identical to the fact-to-face, pre-pandemic cohort (7.04), p = 0.95. Second, for the 
second and third exam that included assessment items that were the target of the two primary 
active learning strategies implemented to improve students' learning outcomes, concept mapping, 
and respectively productive failure, the average mean scores were higher for the online cohort 
when compared to the face-to-face, pre-pandemic cohort (Figure 5). 


Figure 5 
Mean Exam Scores for the Online and Face-to-Face Cohorts 
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While the online cohort had higher mean scores than the face-to-face, pre-pandemic 
cohort, an independent-samples t-test indicated a statistically significant difference for Exam 2, t 
(129) = 9.84, p< 0.001 but no statistically significant difference for Exam 3 (p = 0.30). 

Finally, students' comments from the course evaluation report confirmed the 
effectiveness of the transfer of the course to the online format during the Fall 2020 semester. 
They ranged from focused short answers that recognized this effectiveness, "...thank you for 
easily adjusting class to zoom", to the ability of the instructor to make the course easy to engage 
in, "I enjoyed his class! He [the instructor] was easy to listen to and did very good at keeping us 
focused, which is very hard to do when a class is only online" and to the more detailed 
description of the benefits of this fully online format compared to the regular in-class format: 

Overall, I really enjoyed going to this class every day, even though I do not normally 

enjoy Zoom lectures because of how dry they tend to be. He [the instructor] was quite the 

opposite. He also was able to get at least 90% of the class to turn on their web cameras 
during the lecture, which was impressive. 


Discussions and Further Research 


The research results from the analysis of students' self-reported and assessment outcomes 
data indicated that highly integrated and targeted active learning tasks can be effectively 
transferred to an online synchronous format. The student perceptions, epistemic beliefs, and 
assessment performance were similar, slightly better, or statistically significantly better than the 
ones we found in the live, pre-pandemic format of the course. In addition, student input in the 
final course evaluations supported our findings from the quantitative analysis. 
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The study aims at comparing the effectiveness, popularity and the ease of applicability of different 
learning tools in virtual classrooms among university teachers and students, with reference to the 
users’ technological literacy and trainings, as well as equipment support offered by the universities 
during the pandemic. Comparisons between face-to-face teaching in classrooms and virtual 
classrooms online will be drawn, with respect to limitations, incentives, motivation and 
effectiveness towards learning. This also leads to the question of future course development by 
exploring the possibility of course design and assessment restructuring with a switch to online 
education with the new mode of technology as the trend. 


1. Introduction 
This research conducted aims at investigating the tertiary learners’ and educators’ perception and 
attitudes towards gamification in both online classrooms and face-to-face classrooms, during and 
before the outbreak of the pandemic in Hong Kong. This study also focuses on the effectiveness, 
practicality and limitations of game-based learning in motivating and engaging students towards 
effective and sustainable learning in remote and traditional classrooms in higher education in Hong 
Kong. 

2. Methodology 
Two sets of questionnaires were administrated and distributed to university and college students 
and teachers in Hong Kong studying and teaching in various English Language courses at different 
levels. Among the student respondents in questionnaires and surveys, a vast majority of them 
(90.7%) are undergraduate degree students from different disciplines and specializations taking 
English Language subjects across curriculum among different universities in Hong Kong during 
the pandemic era. More than half of them (63.2%) are currently studying bachelor degree 
programmes in The University of Science and Technology, followed by undergraduates from The 
Hong Kong Polytechnic University (23.7%), City University of Hong Kong (5.3%) and King’s 
College of London (1.3%). 


Apart from education background and the level of competence in English Language acquisition, 
gender also plays a role as determinator of the competence and attitude towards game users in 
classrooms. Considering this variable, respondents are asked to indicate their gender in the 
questionnaires. Interestingly, a majority are males (72.4%) while a minority are females (26.3%). 
In addition, 90.7% belong to the student group whereas 9.3% are English Language teachers in 
universities and colleges in Hong Kong. Further data analysis later reflect how demographic and 
other gender variants can govern the competence of digital technology, practicability of 
gamification and thus the effectiveness and success of gamification in classrooms, which 
influences the level of interactive learning environment and student-teacher relationship in both 
online and face-to-face classrooms. 
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3. Results and Findings 

It is concluded game-based learning is an interactive learning methodology and instructional 
design strategy that integrates educational content and gaming elements, by delivering interactive, 
game-like formats of instruction to learners [2]. Moreover, such learning integrates aspects of 
experiential learning and intrinsic motivation with game applications that have explicit learning 
goals, thereby allowing learners to engage in complex, problem-solving tasks and activities that 
mirror real-world, authentic situations [2]. With simulation and physical artifacts, physical 
classmates could be simulated during the pandemic period. 


3.1. Learners’ perception and attitude towards gamification among universities and colleges 
in Hong Kong 

Among the student respondents, a vast majority (87%) of university and college students indicate 
positive perception towards the effectiveness of learning in classrooms through gamification. On 
the other hand, only 1.4% disagree learning through games is effective. More than one-tenth 
(11.6%) totally agree that learning through games is effective while more than half of them (55.1%) 
strongly agree such teaching strategy is effective. More than one-third (31.9%) believe that it is 
somehow effective. In the view of the perception of effectiveness of gamification towards 
classroom learning from students’ perspective, a majority (73.9%) think that incorporating games 
is a constructive means to learn. 15.9% regard this approach as the most effective, followed by 
58% perceive gamification is very effective in tertiary education and 18.8% are neutral. 5.8% 
disagree that gamification is an effective pedagogy in helping university and college students to 
learn better while 1.4% regard it as the least effective way. 


More than half of them (53.8%) believe the major reason is that learning through games in 
classrooms is fun, followed by 26.2% regard visuals and colours are more appealing than plain 
words which come as the second most important concern. Interestingly, more than one-tenth of 
student respondents (10.8%) explain that they want to win their fellow classmates which 
demonstrates peer influence as an important factor in governing the level of class participation. 
Less than one-tenth (7.7%) feel that gamification encourages them to accomplish tasks through 
teamwork, which in return motivate them to learn with peers. 1.5% believe that playing games 
online has been the trend and therefore it is necessary to incorporate games in classroom learning. 


3.2. Students’ recount of university teachers’ experience in using game-based platforms and 
other innovative digital tools in online classrooms and face-to-face classrooms for English 
Language subjects 

Among all the game-based platforms, more than half of the student respondents (55.4%) recall the 
experience of having their English Language teachers using “Kahoot” in their virtual classrooms 
in their universities or colleges in Hong Kong in both online classrooms during the pandemic and 
face-to-face classrooms before the pandemic. Shared document serves as the second most common 
digital tool in the language classrooms, which constitutes almost one-fifth (18.5%) of students 
have the experience in being asked to use shared document. 7.7% of them have experienced 
roleplays in English Language classes, followed by 4.6% have used storyboards before. A small 
minority (3.1%) have used Class123. The other 3% indicate that they have been instructed to use 
Soqqle and Flipgrid, with 1.5% students out of 3% revealing their English Language teachers have 
used Soqqle and Flipgrid respectively. 5.5% reveal that they have never experienced any digital 
learning tools in classrooms except Zoom as the major communicative classroom platform during 
the pandemic in Hong Kong. 
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[3] points out that teachers experiment different digital teaching strategies which focus on visual 
such as images, paintings and shapes; auditory through rhythms, chants and tones; and kinesthetic 
like body movement and gestures, in order to help students to stay engaged with the taught 
materials in the remote learning. Among all, one of these teaching strategies is learning through 
gamification. With reference to the comparison of frequency of usage of various games in online 
classrooms and physical classrooms, 34.8% of student respondents believe that teachers have been 
incorporating games in online classrooms as much as face-to-face classrooms. While 33.3% claim 
that there are more games to be designed in physical classrooms, 26.1% recall more games have 
been incorporated in online classrooms during the pandemic instead. Only 5.8% cannot recount 
their learning experience in relation to learning through games in classrooms. 


3.3. “Kahoot” is favoured as the most popular game-based platform in digital learning 
among university students and teachers in Hong Kong 

Among all the game-based platforms, it is found that Kahoot is regarded as the most popular game 
by English Language teachers in both online and face-to-face classrooms. It has a user-friendly 
interface that requires low level of technical expertise and the use of “Kahoot” increases 
undergraduate students’ motivation because of its easy-to-use implementation [4]. According to 
research from [4] the before and after tests, “Kahoot” was concluded to be one of the most effective 
digital tools which encouraged creativity and innovation. In the studies, it was found that active 
participation of students stimulates their imagination and their creative capacity to make their own 
tests and learn from those made by their teachers and classmates. It successfully increases students’ 
engagement and motivation to learn and their ambitions for success as it creates a stimulating and 
competitive environment in which students actively participate. In fact, both students and teachers 
can create a positive learning experience in a clear and understandable way using only pictures, 
video and questions to foster an intensely innovative social learning experience. Furthermore, 
“Kahoot” is easily accessible by any device with internet connection, smart phones, tablets or 
laptops. It thus promotes a type of synchronous interaction that encourages real-time collaboration 
and fosters a sense of community, promoting participatory evaluation that favors the development 
of cross-disciplinary skills [4]. Furthermore, “Kahoot” and other similar innovative tools been 
shown to improve students’ ability to grasp the meaning of new information, ask questions, make 
decisions, and draw conclusions that help achieve learning goals and expected outcomes. The 
results obtained in also confirm students positively value the use of this digital platform, which 
can encourage the adoption of these motivating ICT proposals in similar contexts later [4]. With 
regards to the data on the digital competence of learners, it should be noted that the participants 
welcome these online proposals and feel able to master this platform in terms of understanding 
game options, basic instructions and question formulations. More importantly, no specific training 
or complex technical knowledge is required [4]. 


3.4. University students’ comparison of the effectiveness of gamification in online classrooms 
during the pandemic versus face-to-face classrooms before the pandemic 

In this view, there is a significant remark that the popularity of gamification in classrooms lies in 
face-to-face classrooms more than online classrooms, while there is a similar proportion of student 
respondents reckon that teachers use games in online classrooms as much as face-to-face 
classrooms. This illustrates that gamification is not an exclusive approach to be commonly used in 
virtual classrooms by English Language teachers in universities and colleges in Hong Kong only 
during pandemic era. On the contrary, the contexts and localities of teaching are not a factor in 
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governing whether they adopt gamification in classrooms or not, with or without the impacts from 
the pandemic and social distancing measures in the society. Considering a small proportion of 
students fail to recall their learning experience as shown from the questionnaires, the limitations 
of the effectiveness of using gamification in teaching and learning will be further analyzed in later 
section. 
4. Implications 

4.1. The necessity and need for gamification in online university classrooms during the 
pandemic and the possibility of transformation from traditional teaching practice to game- 
based learning (GBL) 


The promotion of game-based learning (GBL) has undoubtedly changed academic environments 
and traditional teaching styles by significantly modifying the roles of teachers and students [4]. In 
particular, GBL implies more active participation in these learning processes with regards to 
students, who responds more effectively to their current interests while improving digital literacy 
and promoting quality and sustainable education [4]. To achieve these objectives, the emergence 
of new teaching and learning models has encouraged educators, as social actors, to adapt to the 
needs of learners in order to create conditions suitable for developing more motivating and 
innovative practices [4]. Today, remote controls are no longer necessary because smartphones, 
tablets or laptops favor the implementation of these systems due to wireless connections, 
applications and websites. Therefore, content knowledge and fun can be merged into daily lessons 
without the need for other intermediate devices due to the advancement and application of 
Information and Communication Technology (ICT). On the Internet, a variety of high-quality 
online platforms can be found such as “Kahoot”, “Socrative’, “Quiz”, “Acadly” or 
“PollEverywhere”, inter alia [4]. 


However, it poses a challenge to discover the dichotomy and contraction between the 
internalization of necessity of gamification in various modes of classrooms by students and 
teachers, and the success of gamification in virtual classrooms versus face-to-face classrooms in 
enhancing the effectiveness of learning and motivations of university students, especially during 
the pandemic in Hong Kong. Some interviewed students hold the view that online games are 
equally popular in face-to-face classrooms to online classrooms. Likewise, games in face-to-face 
classrooms are as much as being adopted by teachers before the pandemic in Hong Kong when 
compared to online university classrooms during the pandemic in Hong Kong. While it may be 
deduced that the trend of incorporating different games in university classrooms to make the 
classes lively and fun is commonplace due to the global pandemic, some students reveal that the 
process of gamification is actually smoother and more effective in face-to-face classrooms even 
before the pandemic when online mode of classes have not been adopted. This conforms to the 
earlier finding that more than one-third of student respondents (34.8%) in the questionnaire 
highlight that their teachers have been incorporating games in online classrooms as much as face- 
to-face classrooms. Similarly, slightly more than one-third (33.3%) recall that more games have 
been incorporated in their formal face-to-face classes in English Language acquisition. While it 
may be expected that gamification is getting more popular in online classrooms as digital learning 
has become the prevailing norm in education across different disciplines and sectors accelerated 
by the global pandemic, games of different categories and pedagogy have been popular in fact for 
long in face-to-face classrooms even before the outbreak of Covid-19. 


Rather than a surge of popularity of gamification in virtual classrooms, only approximately one- 
quarter (26.1%) of the student respondents recount their memory of having more games in online 
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classrooms in their learning experience in universities or colleges during the pandemic period in 
Hong Kong aforementioned. A few interviewed students also conclude that games in online classes 
cannot serve the purpose of attracting students’ attention during class fully, which means that 
gamification is a less dominant and significant driving force in motivating students to learn in 
virtual classrooms since face-to-face presence promises students’ attention span. In addition, it is 
suggested there is a possibility that some university and college students may just simply ignore 
the game playing part in class during online lessons. However, teachers can ensure every student 
is engaging in the assigned games in face-to-face classrooms due to the physical presence. In this 
light, gamification is not the solely effective way to facilitate students’ interaction with their fellow 
classmates and teachers as much as we expect. Games cannot guarantee all students are entirely 
participating in the whole lesson during either online or offline classes. 


4.2. Students’ preference of various gaming tools and perception of the reasons for English 
Language teachers not adopting gamification approach 

Regarding the categories and nature of games to be incorporated in virtual learning, almost half of 
the student respondents (47.8%) claim that they prefer both competitive and collaborative games. 
More than one-fourth (26.1%) prefer competitive games, which compose of the competitive 
elements among classmates. Level-up games in which students need to proceed to different levels 
are only popular among less than one-fifth (18.8%) of the student respondents. Collaborative 
games come to the least popular, in which only 7.2% of university and college students in Hong 
Kong prefer teamwork during class activities. 


It is found that university and college students in general have the perception that time is the 
dominant factor for the absence of gamification in university classrooms. Less than half of them 
(41.8%) believe that teachers lack time to prepare for games to be incorporated in classrooms, 
especially during the pandemic era. However, almost one-fifth (19.4%) perceive their English 
Language teachers are not synchronous with the concurrent trend and thus gamification in 
classrooms is not adopted. 16.4% account their teachers may not understand the importance of 
games, while 10.4% interpret their teachers may not know much about games and 9% believe their 
teachers may not comprehend the importance and need of games. 1.5% indicate the absence of 
games in classrooms is due to the difficulty in designing different games for some specific subjects 
like language, literature and statistics. 


4.3. University students’ perception and attitude towards gamification in general in English 
Language teaching classrooms in Hong Kong 

It is interesting to find out that students in general tend to equalize whether lessons are fun and 
interesting as the determinators for their intrinsic and extrinsic motivation in learning. Most of 
them assess whether their learning experience is effective or not mainly on the basis whether their 
teachers can keep their attention long during the entire classes. In addition, physical sensations to 
physical body and positive psychology both play the vital roles in motivating university students 
to learn better in classrooms. Apart from the fun nature of games as the chief motivator, the second 
most important reason to support games to be incorporated in classroom learning is due to the 
intrinsic nature that visuals and colours in the design of games are more sensationally powerful 
than plain words, followed by learners’ desire to win their peers as the third key cause in their 
conclusion that gamification can motivate them in both virtual and face-to-face classrooms. In 
other words, it is obvious that traditional classrooms with teachers’ one-way lecturing and teacher- 
centered classrooms are no longer enough to satisfy students’ need for innovative learning as the 
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new trend nowadays, especially with the acceleration of digital learning impacted by the outbreak 
of Covid-19 in the globe. [5] identifies gamification as “using game-based mechanics, aesthetics 
and game thinking to engage people, motivate action, promote learning, and solve problems”. 


An active learning context refers to the various learning approaches and instructional methods 
such as experiential learning, collaborative learning, cooperative learning, case-based, inquiry- 
based, problem-based, team-based and game-based learning. These different models cover the 
subset of active learning. Hence, active learning is an umbrella concept that encompasses the 
different learning approaches and instructional methods of learning [2]. As aforementioned, almost 
half (47.8%) of the university and college students respond they prefer both competitive and 
collaborative games, followed by competitive games as the second most popular type of games 
(26.1%), then level-up category of games (18.8%), and finally collaborative games (7.2%) come 
to as the least popular kind. From the collected data, it is illustrated that university students have 
greater expectation on the variety of games that they can be exposed to in classroom learning 
owing to the commonplace digital learning and technological competency. Competitive games in 
which students compete with their fellow classmates and level-up games require players to proceed 
to advanced levels progressively are more popular than collaborative games comparatively. By 
examining the demographic background of the targeted respondents and interviewees, these 
groups of university and college students belong to a more advanced and mature learners than the 
younger learners before the admission to tertiary education. This can be explained by the 
psychological behaviour of learners, who are academically and mentally stronger students from 
dominant universities and colleges in Hong Kong, and simultaneously having the experience in 
surviving through public examinations, tend to be more used to competitive learning environment 
and thus are relatively more self-driven to win. On the other hand, collaborative games such as 
games relate to teamwork are regarded as less popular among these student groups. This can imply 
these student respondents in general may imagine their fellow classmates as academic rivals rather 
than teammates to collaborate and cooperate with. Level-up games are also popular due to the 
sense of achievement and accomplishment that students can attain by proceeding to another level, 
which can also be impacted by the prevailing level-up concept designed in video games that 
students are always exposed to nowadays. 


Many studies have proved that friendships play a critical pivot on students’ social, emotional, and 
cognitive development. [6] found that about 50% of students’ achievement-related comparisons 
were made with their best friends and they often prefer to compare themselves with friends. 
Moreover, some studies indicated that friendship relations are a key role in maintaining positive 
interactions and alleviating negative interactions among students in a learning activity. 
Theoretically, friendship relations are beneficial for students situated in competitive learning 
environments, but friendship relations are still absent in relevant studies, especially on the 
surrogate competition [6]. Apart from friendship, some studies have also shown that gender 
differences can impact the preference over a competitive learning. According to [6], boys are more 
highly motivated to participate in game-based learning environments than girls and tend to have 
higher incentive to attain higher scores in competitive games than girls. This means gender 
stereotypes exist and thus play a role in students’ learning attitudes, which corresponds to the 
demographic background of student respondents which are represented by 72.4% males and 26.3% 
females aforementioned. When students are involved in an effort-demanding activity like 
competitions, they need to improve their learning status to win and thus are guided to realize that 
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winning is closely correlated to the level of effort they exert. This cause-and-effect relationship is 
helpful for the establishment of a positive attitude towards motivational learning [6], which is 
enabled by competitive gams in classroom learning. 


4.4. University teachers’ perception and attitude towards gamification in virtual English 
Language teaching classrooms in Hong Kong during the pandemic 

In order to analyze from a more holistic view in the practice and success of gamification in 
university classrooms in Hong Kong during the pandemic, data from English Language teachers 
is also collected for university teachers’ perspective towards the usage of games in classrooms. It 
is found that a majority of teacher respondents (62.5%) agree that gamification is an effective 
means to motivate students in universities and colleges to learn better, with 25% totally agree and 
37.5% strongly agree that incorporating games in classrooms is an effective approach to motivate 
students to learn better. In contrast, more than one-tenth (12.5%) strongly disagree that 
gamification is successful in motivating students while one-quarter (25%) are neutral towards 
gamification in both online and face-to-face classrooms. 


It is clearly indicated that a majority of teacher respondents (62.5%) hold the belief of the necessity 
of positive reinforcement towards student motivation with the use of gamification in both online 
and face-to-face classrooms among universities and colleges in Hong Kong. However, the 
comparative findings between student respondents and teacher respondents also illustrate 
distinctive and different perceptions and attitudes towards learning through games among the two 
groups. As shown from the data analysis aforementioned, a majority of students (73.9%) think that 
incorporating games is a constructive means to help their effective learning in English Language 
classrooms, with the major concern about whether classroom learning is fun and interesting, which 
constitutes more than half of student respondents’ (53.8%) need as learners for second language 
acquisition. In addition, almost all students (94.2%) claim that games act as remarkable incentive 
to draw their attention in classes of different modes and thus drive them to learn dependently inside 
and independently outside classrooms. In contrast, viewing the teachers’ attitude towards the 
incorporation of games in both virtual and physical classrooms, more than half of university 
English Language teachers (62.5%) internalize the importance of gamification and other digital 
teaching tools as the current and upcoming innovative trend and thus there is a need to adopt a 
more interactive approach for students by devising new pedagogy in relation to digital technology 
in order to supplement or even replace one-way lecturing in traditional classrooms. According to 
[4], systems that only focus on lecturing and other traditional teaching strategies produce passive 
learners. “Spoon feeding" techniques in traditional classrooms tend to suppress students’ creativity 
and neglecting their strengths as students are highly dependent on their teachers’ lecturing instead 
of independent thinking [4]. One-way lecturing also demotivates students to learn effectively if 
lessons are found to be dull and boring. In general, students prefer games to be incorporated in all 
classrooms more than teachers, given the intrinsic nature of collaborative and competitive games 
as motivators in getting students to work with peers, with 31.7% difference regarding the 
popularity of game usage between university students and teachers. 


The statistics also demonstrate that all teacher respondents have the experience in running their 
English Language classes with gamification, as shown from the questionnaires that none of any 
individual teacher respondent claims he or she has never used games as teaching tools in university 
classrooms. Nevertheless, when it comes to the practicality and frequency of the usage of games 
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in English Language contexts, only half of them (50%) incorporate gamification in classrooms 
generally for a few times every semester for English Language teaching. If there are 13 to 14 weeks 
in every semester, it is implied that only approximately 15% to 35% of their total class time have 
been devoted to the practice of gamification in university classrooms. One-quarter (25%) recount 
that they include the elements of games in their English Language courses for more than half of 
the semester, which means this group of teacher respondents have spent half of the class time on 
incorporating games in university classrooms for students. Meanwhile, only more one-tenth 
(12.5%) design their English Language classes once every week and the other 12.5% conduct 
games in classrooms once every two classes respectively. 


While a vast majority of university teachers (87.5%) reckon incorporating games in classrooms of 
any form is constructive in helping students to learn more effectively and positively, it is indicated 
that there is also a majority of 75% teacher respondents believe games motivate students to 
participate more fully in class and learn better. On the contrary, 12.5% of university teachers think 
gamification is not effective for students to learn better while 25% regard gamification cannot 
motivate students. It is concluded that there are striking differences between learners and educators 
in universities and colleges in Hong Kong, with reference to students and teachers’ perceptions 
and attitudes towards the necessity and the effectiveness of gamification towards effective learning 
and motivator of learning. 


4.5. University teachers’ comparison of the effectiveness of gamification in online classrooms 
during the pandemic versus face-to-face classrooms before the pandemic 

In terms of effectiveness in using games online and face-to-face, most interviewed teachers believe 
that student engagement is the major concern. It is easier for teachers to engage with students in 
face-to-face classrooms due to their physical presence in concrete settings. On the other hand, it is 
harder to build rapport with students when the lessons go online, making teachers’ assessment of 
students’ learning progress more difficult. While students can switch off cameras on Zoom in 
virtual classrooms, teachers can walk around in the face-to-face classrooms to establish a sense of 
presence. Thus, implementation of gamification becomes more inclusive for each student in face- 
to-face classrooms, in which both active and passive learners are more motivated by gamification 
in physical classrooms, making learning more direct and effective. Nonetheless, there is a 
possibility that gamification can exclude the passive learners in online classrooms since it is more 
challenging for the teachers to supervise every student online. Meanwhile, a few interviewed 
teachers advocate that the ease of incorporation of games in online classrooms and face-to-face 
classrooms are both at a similar level. The most dominate reason relates to the positive psychology 
among university students that learners are in general attracted by the nature of games and fun 
lectures. Furthermore, the content of the lecture is always very much the same and materials are 
delivered via the similar platform. 


4.6. Limitations of gamification in remote and face-to-face classrooms Most of the student 
interviewees hold the view that there would be differences in respect to the effectiveness in using 
games in online in face-to-face classrooms. Most of them reckon that gamification is more 
effectively implemented in face-to-face classrooms before the pandemic rather than virtual 
classrooms during the pandemic in tertiary education in Hong Kong. Interestingly, it is generally 
believed that learning through collaborative games is more effective in face-to-face classrooms 
than virtual classrooms as shown from the fact that most student interviewees internalize the notion 


345 


that face-to-face classrooms promote collaboration and effective communication in comparison to 
remote learning. Apart from collaborative games, it is also stated that competitive games online 
may be less fun and interesting than games in face-to-face classrooms, explained by the difficulty 
of online classrooms in establishing a competitive environment to encourage students to engage in 
competitions. Comparatively, face-to-face communication enables learners to interact and 
accomplish given tasks collaboratively in an easier and more comfortable way, providing the 
concrete physical settings which allow spontaneous and direct communication instead of potential 
communication barriers in intangible virtual channels. Hence, face-to-face classrooms facilitate 
both collaborative and competitive games more effectively in helping and motivating students to 
learn better than online classrooms. 


In addition, online classrooms pose a challenge for teachers to create a positive and proactive 
learning atmosphere and thus is more difficult to set up the mood for game playing in virtual 
classrooms. Furthermore, the difficulty in implementing gamification to all students in online 
classrooms without excluding any passive learners is a crucial hinderance for student engagement. 
This includes the constraint in monitoring whether students are following house rules during 
online classes since some students can possibly turn off the cameras and mute themselves. One 
interviewed student cites an allegory that lecturers can spot out any student in physical classrooms 
who is not paying attention and thus the internalization that students should listen and respect the 
lecturers can in return facilitate students to concentrate better and reinforce students’ full 
participation in face-to-face classrooms. One interviewed student has internalized the idea that 
online learning is dull and boring anyway and he is always distracted at home especially when he 
is sitting in a comfortable private area. It is generally believed among university students that they 
and their fellow classmates are more proactive in face-to-face lessons. This corresponds to some 
interviewees who proclaim that learning through games would be more “interactive, exciting, fun 
and attractive” with face-to-face and direct communication since the sense of student engagement 
increases behaviorally and emotionally in physical classrooms. It is also said that face-to-face 
classrooms can trigger more interactions between teachers and students and among students, which 
result in better student-teacher and student-student rapport. 


On the other hand, the technological nature of virtual classrooms hinders the effectiveness of 
incorporation of games in digital learning experience. Among the interviewed students, it is found 
that they sometimes experience delay of response from teachers and classmates due to internet 
connection problem in remote classrooms, and thus reckon face-to-face learning to be more 
effective and direct. 


4.7. The gap between university teachers’ acknowledgement of the vitality of gamification in 
classrooms and the frequency of its usage in practicality 

Given most university teachers recognize gamification as a constructive, effective and innovative 
means in facilitating students’ learning, it is indicated that only 12.5% of teacher respondents 
incorporate games in their English Language courses once every week and once every two classes 
respectively. Likewise, none of any teacher respondents disagrees that gamification in classrooms 
is effective in motivating students to learn. The limitations of the practicality of gaming in both 
online and face-to-face classrooms can be justified by numerous obstacles in adopting gamification 
in university classrooms and other digital teaching tools. A large majority (75%) of teacher 
respondents hold the view that one major hinderance in incorporating games in classrooms is due 
to the tight teaching schedules. Half of them (50%) believe that there is lack of trainings and 
resources received and somehow games and the content of courses are not interrelated. The third 
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most common limitation (37.5%) is that there is lack of access to different software and tools to 
support the learning through games in classrooms. It is unavoidable that the use of realia and 
concrete props in classrooms are unlikely in virtual classrooms during the pandemic. Therefore, 
implementation of gamification in online classrooms requires more institutional support, trainings 
to be received and know-how of the technology than that in the status quo, which further limit the 
feasibility and scale of incorporating games in classrooms during the pandemic. 


4.8. Effectiveness and practicality of incorporating games in online classrooms versus face- 
to-face classrooms 

Regarding the kinds of innovative teaching tools used in classes so far, most interviewed university 
teachers name a few including “Zoom polling”, “Zoom whiteboard”, “Kahoot”, “Padlet’”, “Word 
Clouds”, “Everything Poll”, “graph Drawing”, and “Lucky Draw”. It is also highlighted that 
“Kahoot”, “Padlet”, “Spotify”, “Word Clouds” and “Lucky Draw” are their preferred tools in both 
online and face-to-face classrooms for English Language teaching in universities and colleges in 
Hong Kong. From the results of interviews of teacher respondents, it is revealed that university 
teachers who deploy the concept of gaming in classrooms are in the purpose of enhancing 
spontaneous interaction and participation among students. It is also relatively easy for both 
teachers and students to use these game-based tools. On top of the interactive nature of gaming in 
classrooms, concepts can be visualized to students more easily and effectively with colours, 
symbols, graphs, shapes, videos and music through games. 


In addition, the interviewed teachers hold the belief that games are catchy to students’ attention 
and teachers can make use of gamification to monitor whether students are attentive in class at that 
time. However, one interviewed teacher holds an opposite view by raising her concern that there 
is a possibility that some students may feel harassed by the compulsory gaming in classrooms and 
students may also believe learning should be serious instead. Most interviewed English Language 
teachers explain the usage of games in classrooms can be hindered by the fact that the application 
of games and other digital tools rely too much on the content of the subject knowledge at that time. 
At times, ideas and inspirations from games can be interrupted by the lack of originality. Besides, 
the unstable internet connection for teachers and all students makes smooth adaptation of games 
in online classrooms during the pandemic even more difficult. 


While gaming tablets and social media have been a major part of students’ life, teachers face a 
significant problem towards students’ motivation and achievement inside classrooms. Learners get 
easily distracted and show a loss of interest, and communication between students and the teachers 
becomes remote and fragile, especially with the distancing learning during the hit of pandemic [4]. 
In particular, some teachers even find the digital platforms discouraging, as they have to do extra 
work to adjust the pace of the class to achieve a better understanding of the content. This requires 
enormous effort both inside and outside the classroom in order to integrate the digital content into 
lesson plans. This also requires the intensification of educational adaptations made in classrooms 
to promote content learning by all students in a comprehensive and meaningful way. Another 
disadvantage pointed out in some studies is linked to the negative attitude of some students to these 
digital challenges, since not all students prefer to play an active role in classrooms [4]. In fact, 
some of them feel more comfortable taking notes and studying content after class without using 
their mobile phones for academic purposes because they fear making mistakes in public when 
using this digital resource or not feeling supported by their peers when asked about content 
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previously worked on in public [4]. Further research and information on the application of these 
innovative proposals in higher education contexts is therefore needed to better understand and 
adapt these ludic strategies to the main interests and demands of students in higher education. 


5. Conclusion and Limitations 

This research compares and evaluates the success of gamification and effectiveness of various 
digital teaching and learning tools used in motivating students in university classrooms in Hong 
Kong during the pandemic. It also explores the possibility and limitations of applying the notion 
of gamification in virtual classrooms. With the research analysis, gamification in classroom 
learning has been an unavoidable trend in education. With data collected from both learners and 
educators towards English Language learning in tertiary education in Hong Kong, the future 
research can be extended to compare the data analysis from both pre-and post-tests, with the 
integration of different games for experiments into the teaching process to assess the level of 
students’ active participation and motivation towards a more interactive and stimulating 
environment. It is also recommended that more resources, trainings and technical support can be 
offered by the authorities and educational institutions to pursue the sustainable game-based 
learning and provide a more engaging and interactive environment for learners in the new 
technological era. 
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Abstract 


Faculty and graduate students in an Educational Design and Learning Technologies program 
collaboratively developed their own online wisdom-community in Canvas as a project in a Foundations of 
Learning Design course. Community goals established by program faculty and students were to facilitate learning 
community, enhance identity as professional educational technologists, expand professional knowledge for the 
duration of their studies toward their degrees, and leverage and prepare members for use of various online learning 
technologies. We analysed and explored the EDLT learning context, learners, and goals, and student-constructed 
prototype environments following educational design and development research methods. Findings were needs, 
tasks, and features students wanted addressed in their WisCom, as well as suggestions for implementation. 


Imagining something may be the first step in making it happen, but it takes the real time and real efforts of real 
people to learn things, make things, turn thoughts into deeds or visions into interventions. — Mr. Rogers 


Problem 


Masters’ and doctoral students often do not mingle or have opportunities to learn from and mentor each 
other. Instead, they attend classes face-face or online, study, and complete assignments and assessments 
independently. Group work provides them with opportunities to work with and learn from each other, but such 
work limits their exposure to a small group of people in the same courses in which each is enrolled. Students do not 
have the benefit of learning from others across the spectrum of experience in their degree programs, particularly if 
they are fully online and are unable to mingle in the brick-and-mortar college setting. 

For several years, in the hope of creating bonds among masters’ and doctoral students and because of the 
structural, social, and organizational benefits cohorts promise (McCarthy, et al., 2005), Educational Design and 
Learning Technologies (EDLT) students have been accepted into their programs at our Southwestern, Hispanic 
Serving Institution (HIS) as members of cohorts. In our cohort model, students’ programs are prescribed in that all 
members begin the program together and progress through course content in a shared sequence and pace. 

However, the cohort model has created some problems. Learning in cohorts means that students are all 
studying the same content at the same time. Those who do not have time or finances to attend a full-time degree 
program may choose not to apply. Each cohort consists of just those students who enroll in the program in the same 
term. Following the prescribed course-sequence limits students’ abilities to receive differentiated, personalized 
instruction. Students rarely form solid bonds or a shared knowledge-base, and faculty members are detached from 
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most students other than those that they advise or intermittently instruct. McPhail, Robinson, and Scott (2008) 
reported that factors such as dominant group members, lack of commitment to the cohort, failure to meet group 
expectations, traditional instructional modalities, and inadequate facilities negatively impact the cohort experience. 
Most importantly, the cohort model limits enrollment to traditional students who can afford the time and resources 
to participate in the scope and sequence of a highly structured degree-program. 

In short, the current structure of college degree programs in general can limit students’ abilities to network 
with other students in order to learn from the strengths of diverse others who have different perspectives, cultural 
backgrounds, and levels of experience and professional knowledge. As Gunawardena, Frechette, and Layne warn 
(2019), “Without proper deliberation, online learning experiences...[can] unduly reflect the cultural biases and 
limitations of their architects,” (p. 1). 


Theoretical Framework, Previous Research, and a Solution 


Sociocultural learning theories provide a rationale for building any online learning community that 
facilitates discovering content knowledge, solving problems, thinking critically, forming identity, and including 
voices of diverse learners across their degree seeking experiences. The sociocultural perspective on learning 
emphasizes that learners develop and learn by transforming their understandings through socially shared activities 
conducted with diverse others. Learners with diverse levels of competence learn from one another as well as from 
their instructors (Vytotsky, 1978). Each learner has unique knowledge, needs, experiences, culture, and 
expectations that, when shared, can broaden others’ perspectives and knowledge bases while they themselves 
reciprically benefit from others. A learning community of students within and across degree programs in a specific 
discipline can provide a venue for students’ sociocultural learning by exposing them to a broad population of other 
learners who have diverse interests, experiences, and circumstances. 

Although socio-cultural learning theories have been proposed for many years, developing learning 
environments that facilitate sociocultural learning has been difficult until the advent of online learning technologies. 
Such technologies can bring diverse thinkers, experts, and learners together in distributed learning communities to 
contribute to each other’s learning. 


Online Learning Communities 


Students can both learn and gain personal satisfaction in online learning communities when they fully 
participate in them. Much has been written about the power of interaction, mentoring, and presence between 
students and faculty in online learning communities. In addition, online peers have tremendous influence on one 
another. Student-to-student interaction can lead to increased levels of student satisfaction and student learning 
outcomes (Eom & Ashill, 2016). The Association of American Colleges and Universities identified establishing, 
building, and maintaining learning communities as a high-impact practice that leads to student success (Brownell, & 
Swaner, 2010). 

The literature on learning communities most typically focuses on creating online community in the context 
of coursework. But, community rarely happens in the context of a single course. Rather, community can be formed 
at the program level by intentionally coordinating and linking the content of courses, materials, assignments, grading 
rubrics, and course resources within programs; orienting students to expectations across a program; using 
instructional-teams; and using engaging pedagogies (Brownell, & Swaner, 2010; Linder & Hayes, 2018). As Jody 
Donovan (2015) claims in her blog, “taking an online course should be more than sitting in front of a computer — 
real engagement involves becoming a part of the community of learners.” Learning communities can provide 
diverse college students with a sense of belonging to a group that shares their goals and interests. Often individuals 
in shared communities interact through social media beyond their courses and become colleagues as they build their 
careers. 

Palloff and Pratt (2011) created a framework for distance learning that generates growth of learning 
community in online programs. Their framework advises that— 

- online [communities] should include focused outcomes with buy-in from everyone in a program and time spent 
sharing goals, 
content knowledge should be achieved actively through and with interaction and feedback, 
[Communities] should include facilitated collaboration, and, 
faculty guidance toward teamwork with mutually negotiated guidelines helps students be part of a learning 
community. 
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The hope of online learning communities is that they increase comfort, communication, and collaboration 
among students and with instructors. Online collaboration tools and social media can be incorporated into online 
communities to promote learners' senses of community and increase the knowledge flow between students, thereby 
facilitating social negotiation of meaning as learners construct their own understandings (Bliss & Lawrence, 2009; 
Dawson, 2018; Kumi-Yeboah, 2018). 


Wisdom Communities 


In a wisdom community (Gunawardena, Frechette, & Layne, 2019), here forward called a WisCom, 
students use technologies to communicate with one another online. Social interaction, dialog, discourse, 
collaborative problem-solving, and construction of new knowledge with instructors and peer guidance are the 
fundamental activities of WisComs. Gunawardena et al. call this transactional approach “distributed co-mentoring.” 

An alternative to requiring students to study within a cohort is to offer an online WisCom that spans the 
duration of students’ programs-of-study. With the goal of facilitating bonding and shared knowledge, students bring 
their cultural and historical perspectives, experience, and knowledge to each other and form bonds around a shared 
identity. Social interaction, dialog, discourse, collaborative problem-solving, and construction of new knowledge 
with instructor and peer guidance are the fundamental activities. Co-mentoring, and learner support play critical 
roles in wisdom-communities. Communication, distributed co-mentoring, and learner support take place within 
“collaborative inquiry cycles.” Members work together in collaborative inquiry cycles (CICs), one cycle at a time, to 
“explore a problem or issue, brainstorm solutions and considerations, and work together to synthesize findings” 
(Gunawardena et al., 2019, p. 278). Once learners agree that the cycle is completed it is preserved and the group 
moves on to the next CIC. 

Along with co-mentoring, learner support plays a critical role in WisComs. Student retention, motivation, 
identity formation, academic achievement, satisfaction, engagement, and success all hinge on learners knowing that 
they are supported (Mehran & Mahdi, 2010). Therefore, a WisCom includes access to interactive activities and 
services intended to support and facilitate the learning process of each student (see Figure 1). 


Figure 1 


Wisdom-Community Framework Created by Casey Frechette for Gunawardena, Frechette, & Layne (2019). Used 
with permission from the authors. 
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A Solution 
In the hope of strengthening the EDLT graduate programs in a Southwestern United States, Hispanic 


serving university, students in a Foundations of Learning Design course designed and developed a WisCom as 
their term project. The WisCom will be used and contributed to by EDLT faculty and Master’s and Doctoral 
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students across the scope and sequence of their studies. Community goals established by faculty and students are to 
facilitate professional learning community, 
enhance sense of identity as members of the global professional learning community, 
expand professional knowledge among community members, and 
e leverage and prepare members for use of various online learning technologies. 

During students’ design and development processes and based upon their final products, we asked the 
following question— Given the opportunity to design and develop a WisCom in the Canvas Learning Management 
System what needs, tasks, and features did students want addressed in their WisCom, as well as what suggestions 
did they have for implementation. 


Methods 


Instructional design methods were applied in this educational design and development research with the 
goal of developing “theoretical insights and practical solutions in real-world contexts, together with stakeholders” 
(McKenny and Reeves, 2019, p. 6). Iterative phases of— 

(1) needs analysis and exploration, 

(2) design and construction, and 

(3) evaluation and reflection comprise such studies. 

In this study, we report on phases one and two. In phase one as suggested by McKenny and Reeves, we attempted 
to “generate a clear understanding of the problem and its origins as well as specification of long-range goals” (p. 
85), for the wisdom-community. In phase two, through teamwork, communication, and creativity we produced a 
potential solution to the stated problem by creating a WisCom in Canvas. In phase three, which is yet to come, we 
will evaluate the impact of the WisCom on faculty and students. We included EDLT students as designer- 
developers following the principles of user design (Carr-Chellman, 2006), also known as participatory design. User 
design involves input from potential users of the design so that the resulting instruction meets their needs. As 
potential users of an EDLT WisCom, EDLT students participated in the design and development of their own 
WisCom. 


Context 


In the context of the graduate level Foundations of Learning Design course, EDLT students helped design 
and develop the EDLT WisCom in accordance with the principles of instructional design. They analyzed, designed, 
developed, implemented, and evaluated example EDLT WisComs. As potential end-users, they were actively 
involved in the design process to help ensure the resulting WisCom would be compelling, usable, and responsive to 
their cultural, emotional, spiritual and practical needs. Recent research suggests that designers create more 
innovative concepts and ideas when working within a co-designed environment with others than they do when 
creating ideas on their own (Treischler, Trischler, J.; Pervan, S. J.; Kelly, S. J.; & Scott, D. R., 2018). Therefore, 
they built Canvas-based WisComs in four teams of three to four students. 

Topics of modules in the course follow: Becoming a Learning Designer, IDer, and Educational 
Technologist; History of the Field; ID Models; Foundational Theories; Needs and Learner Analysis; Task Analysis 
and Identification of Types of Learning; Assessing Learning; Development of Strategies that Address What We 
Know about How People Learn; Implementation and Management of Learning Design Projects; Evaluation; and 
Conclusion. Readings included Culturally Inclusive Instructional Design by Gunawardena, Frechette, and Layne, 
(2019) and the instructor of generated content in each of the modules. The overarching assignment was for students 
to follow the principles of instructional design to develop an EDLT WisCom. Beginning with the fifth module, 
each had assignments that led to the systematic development of their team’s WisCom. 


Participants 


Participants in phases one and two were one faculty member (the lead researcher), one doctoral student, 
and fifteen students enrolled in an eight-week online graduate level course on learning design. The faculty member 
identifies as a white female from the U.S Westcoast with several years of teaching experience in educational 
technology and instructional design. The graduate student identifies as a white female from the Southwest with 
several years of teaching experience and educational administration experience. Students’ ethnicities were 8 
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Hispanic, 4 White, 1 African American, | Native American, and 1 West Indian; genders were 12 females and 3 
males; and location when growing up were 8 from the Southwest United States (U.S.), 4 from the Westcoast of the 
US., 1 from the midwest, | from the South, and | from the West Indies. In terms of teaching experience, 62% of 
the participants had 0-5 years, 31% had 6-10 years, and 7% had 11-15 years. 


Data Sources 


Data sources included 1) a Pre-Wisdom-Community Design and Development Assessment developed and 
administered by the researchers to determine students’ perceptions regarding their needs, 2) a needs assessment 
developed and administered by students and distributed to EDLT students who were not in the class; 3) a goal/task 
analysis developed and conducted by students; 4) assessment criteria identified by students; 5) strategies identified 
and described by students; 6) implementation strategies identified by students; 7) one-on-one and small group 
evaluations conducted by students, 8) a Post-Wisdom-Community Design and Development Assessment, and 9) the 
four student-developed WisComs. 


Procedures 


The fifteen participating students enrolled in the online Foundation of Learning Design course. In a Zoom 
mediated course orientation, students were introduced to their term project of developing an EDLT WisCom for 
future use by Masters and Doctoral students. They were also given written instructions for the assignment in the 
first Canvas module. They were divided into four groups and each group was given a Canvas shell in which to 
build an EDLT WisCom. The Pre-Wisdom-Community Design and Development Assessment was administered 
online using Google Forms in the context of the first course module. It was designed to establish whether or not 
there was a need for a WisCom, whether or not EDLT students were likely to participate in a voluntary WisCom, 
and what topics students would be interested in exploring together in a WisCom. 

In the fifth course module student-teams developed and conducted a needs assessment of their own by 
sending surveys to all EDLT graduate students. In response to subsequent modules, the teams went on to conduct a 
goal analysis, establish assessment criteria, develop WisCom prototypes, describe implementation strategies, 
conduct one-on-one and small group evaluations, and revise their WisComs accordingly. The resulting products 
were four WisCom prototypes. The researchers identified the most effective and compelling components and 
features of each in order to assemble one WisCom that addresses EDLT students’ needs. In the final course module 
students filled out the Post-Wisdom-Community Design and Development Assessment using Google Forms. 


Data Collection and Analyses 


Inputs explored were needs, tasks, and features identified by students as they developed their WisComs, as 
well as what suggestions they had for implementation. Data collection took place in the context of course activities 
in the Foundations of Learning Design course. Therefore, design and development tasks were realistic in scope. 
Multiple data sets were used to triangulate the data. Students collaboratively generated and submitted design and 
development documents for each design and development phase. All data sources were qualitatively focus-coded 
according to the following codes: needs, goals, objectives, assessments, strategies, implementation, evaluation, 
technology, communication, distributed co-mentoring, learner support, collaborative inquiry cycles, WisCom 
features and emergent themes. Those findings will be shared in a subsequent manuscript. 


Results 


The EDLT students strove to gain a clear understanding of the problem they were addressing with their 
design. The tasks were difficult for them, particularly assessment, given that participation in the WisCom 
environment will be optional and content will be developed by students as they participate. They concluded that 
testing or quizzing EDLT students on content in the WisCom context would be oppressive and turn students off to 
participation. They did design exit surveys to collect students’ responses to the environment for formative 
evaluation purposes. They realized that they were to develop a flexible framework for participation. However, their 
final products did not reflect full understanding that co-mentoring and collaboration are to take place across time, 
content, and activities and not in just one space. Also, students did not address content of collaborative inquiry 
cycles or how they would be administered and implemented. We conclude that their avoidance of addressing this 


353 


important feature in WisComs was due to their lack of understanding of how they might be presented and 
sustained. 

The Pre-Wisdom-Community Design and Development Assessment as well as the student administered 
needs assessment that went to all EDLT students established that students felt the need for an online environment 
where they could share content with other students in EDLT. Many offered that they were likely to participate. 
Needs, tasks, and features of the four student-developed prototypes are illustrated in Figure 2 below: 


Figure 2 


The Home Pages of Four Student-Developed Prototypes. 
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WisCom 3. 


Wisdom Community — Group 3 


Modules 1-4 have objectives, specific topics 
to explore, and a quiz. 
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The researchers will identify the most effective and compelling features of each in order to assemble one WisCom 
that addresses EDLT students’ needs, provides for co-mentoring, and facilitates completion of collaborative inquiry 
cycles. The WisCom will be launched in January, 2022. 

Regarding implementation, students offered that systems need to be in place for providing access and 
knowledge of the WisCom to all EDLT students. They suggested that, because participation will be optional and 
students are typically working full time while going to school, motivation to participate must be addressed through 
the design. We will apply the ARCS-V (Keller, 2010; Keller & Deiman, 2012) motivation model across the 
environment and in that context demonstrate the model to students suggesting that they apply it as they lead others 
through CICs. Students recommended using a lot of visual representation to gain EDLT students’ attentions to 
activities. They do not want a text-heavy environment. With each CIC, we will emphasize relevance, confidence, 
and satisfaction. We expect that these emphases will increase students’ volition to be active in the WisCom. 

Students want to run the WisCom themselves indicating that the site should be managed by faculty or a 
graduate student intern under faculty supervision, but that students should generate content. They suggested 
monthly activities. Although the framework proposed by Gunawardena et al. suggests that participation in CICs 
continue until a CIC is complete, for ease of management, each CIC in our WisCom will be led by a small team of 
1-3 doctoral students and last one month at which point a new team will take over. 

Topics of CICs will be chosen by EDLT students. Students will lead and conduct CICs in modules and 
students will know where on the site they are collaborating each month through announcements made by the 
student team that is in charge for that month. Each CIC will begin with an orientation and statements of objectives. 
In addition to contributing to the Canvas WisCom site, students want to share links to valuable content. They hope 
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to work together and network with others in the field using social media and tools such a TEAMS and Discourse, 
so links to those communication channels will be posted. CIC content will be preserved on the Canvas WisCom so 
that students can review previously submitted content. 


Conclusion 


In summary, EDLT students identified needs they wanted addressed, tasks they wanted to work on, and 
features they wanted included in their WisCom in order to collaboratively gain professional identity, knowledge, 
and community during their graduate degree programs in EDLT. The completed first and second phases of this 
longitudinal study set the stage to generate guidelines for developing and implementing WisComs offered over the 
duration of learners’ studies in degree programs. Prior studies have explored the impact of design components in 
courses offered as WisComs. This study explores the impact of a longitudinal wisdom community to inform both 
theory and practice for preparing professional educational technologists. The study contributes to the literature by 
describing how WisComs can be collaboratively designed with potential users to enhance professional identity, 
community, and professional knowledge for students across the span of their Educational Technology degree 
programs. In addition, specific components and features that appealed to graduate students were identified. 
Findings in this study contribute to instructional practice by providing a practical solution to the problem of 
graduate students’ sense of isolation and lack of identity as experts in their fields, particularly those who are 
studying in online programs. 

Ruja Benjamin (2020) tells us that “Emancipatory designs are not only possible, they already exist.” The 
EDLT WisCom is meant to be an emancipatory design that will promote cultural inclusivity in EDLT graduate 
programs. This study validates Gunawardena et al’s framework and guide for building online wisdom communities 
by demonstrating the ease with which the guide can be used by designers and developers (Richey & Klein, 2014). 
In this case, students were able to use the guide in combination with systematic instructional design processes to 
develop a learning environment meant to meet their needs as well as the needs of others. 

A first limitation of the study is that EDLT students designed and developed the WisCom in the context of 
an EDLT course led by a professor who openly shared her vision for the learning environment. Although she 
clearly stated that she was open to all design ideas, she likely influenced many of the students’ design decisions. 
For instance, one of the Foundations of Learning Design modules was on the history of instruction design. That 
students chose to include history in their WisComs may reflect the professor’s value of history. Secondly, the 
compiled specific components and features that appealed to graduate students in this study cannot be generalized to 
other contexts. Rather they are perhaps a jumping off point for exploring the needs of other learning technologies 
students. In this case, students in the specific context of the region and university culture, socially constructed their 
WisComs and each WisCom was remarkably different from the others. 

This study of user design and development of the EDLT WisCom is the beginning. In a future 
representative field evaluation study (Richey & Klein, 2014), we will implement and evaluate the WisCom with 
educational technology students from the time they enter the degree program until they graduate, we will ask the 
following questions: What are the impacts and effects of the WisCom on goal achievement? How has the WisCom 
impacted professional identity, community, professional knowledge, and transformational learning? What 
unanticipated needs arise? What technical, temporal, physical, transactional, or pedagogical components can be 
revised, added, or deleted to better support communication, co-mentoring, collaboration, inquiry, reflection, 
negotiation, and learning among EDLT graduate students? And, how does the WisCom impact EDLT faculty? 
Answers to these questions will inform both socio-cultural learning theory and practice in higher education, as well 
as contribute to the research on online learning communities and, in particular, wisdom-communities. 

Other studies such as this one need to be conducted exploring design and development of WisComs and 
the efficacy of wisdom communities to facilitate culturally inclusive instructional design for students in different 
disciplines, at different ages, and across and within different global regions. 
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Abstract 


This paper explores virtual teaching and learning innovations in Facilitated Study Group (FSGs) 
delivery at the University of Toronto Mississauga (UTM). To increase access and representation 
in an already successful in-person Supplemental Instruction (SI) program, we seized the 
opportunity during the COVID-19 pandemic to create new alternatives by introducing Virtual 
Facilitated Study Groups (VFSGs). We have since introduced hyflex options in peer-led programs 
that include a combination of both face-to-face (F2F) and online SI Instruction. With new format 
options in our program, we recognized the need to re-examine and reflect on the metrics used for 
measuring outcomes and impact. Supported by the Institute for the Study of University Pedagogy 
at UTM where SI is administered, a “virtual cineplex” FSG platform was developed to offer a 
unique and efficient access point for students to network, congregate, and self-select new 
variations and opportunities in FSG programming. Prior to the pandemic, a specialized 12-week 
undergraduate experiential academic course, EDS325: Supplemental Instruction in Higher 
Education, had already been introduced to enrich and deepen the training of FSG leaders (FSGLs). 
The course is experiential in nature and requires a 100 hour internship in FSG instruction. This 
one-of-a-kind for-credit course engages FSLs in SI theory and scholarship while supporting post- 
graduate outcomes with a focus on transferable leadership skills and professional learning. In the 
winter of 2020, EDS325 made its debut as an online course and is now offered in both virtual and 
in-person formats. This paper tells the story of several lessons learned in hyflex program expansion 
as we reimagine the multimodal and multiplatform future of SI. 


INTRODUCTION 


Supplemental instruction (SI) was introduced across university campuses in the early 1970s as an 
alternative approach to academic instruction. A voluntary offering, students participate in peer-led 
sessions that meet outside of class time in small subject and course-specific groups to work and 
study collaboratively. Post-secondary institutions have historically tied evaluation of SI to student 
enrolment and specific performance metrics, most often looking for improvements in GPA and 
university retention. However, as Vijay et al. explains, “It is critical both to identify at-risk students 
and implement evidence-based instructional strategies and interventions [...] with recent literature 
showing that cognitive, affective and demographic characteristics all contribute as risk factors” 
(2021; p.552). Alongside recent hyflex opportunities in SI, we propose the use of alternative 
metrics that also focus on the “affective” domain to better understand FSG experiences and, in 
particular, FSG leader experience. Focusing new metrics around how FSG leaders experience 


358 


their role fills an important gap in research around the potential for SI impact. We decided to look 
at affective characteristics cultivated through the frame of Debebe’s “psychological capital” 
(PsyCap) criteria described in the acronym “HERO” - Hope, Efficacy, Resilience and Optimism 
(2017). With a robust analysis and correlative study on leaders, we can learn more about hyflex 
SI programming and its potential. 


In response to the COVID-19 pandemic, the University of Toronto Mississauga (UTM) 
successfully launched Virtual Facilitated Study Groups (VFSGs) that served 2,409 students 
through the instructional efforts of 271 Facilitated Study Group Leaders (FSGLs) supporting 64 
undergraduate virtual non-credit courses over the Fall 2020 and Winter 2021 academic terms. By 
moving UTM’s existing Facilitated Study Group (FSG) program from in-person to hyflex we 
expanded our ability to offer alternatives as students sign up for a combination of in-person and 
virtual learning opportunities. During development, we wanted to ensure the level of access and 
efficacy in this already well-established and successful program would be continued. Seizing the 
opportunity during the pandemic to move forward with the offering of hyflex alternatives in SI, 
we sought to focus more specifically on the experience of virtual FSGLs as they adjusted to online 
SI Instruction on our campus. Acceleration in SI program-level innovation took place through a 
trial-and-error process with our FSGLs helping to remove instructional restraints while “shaking 
loose” important new research questions to inform SI theory and practice. The following paper 
will share our progress and some key milestones fostering transformative learning experiences in 
FSGL training. 


Reassessing Metrics: Psychological Capital (PsyCap) as a Metric for Impact 


First developed at the University of Missouri-Kansas City in 1973, SI in its inception was 
structured as an in-person, peer-to-peer, and co-curricular academic support program. Gaining 
considerable momentum through the 80s and 90s, students on college and university campuses 
recognized the value of engaging in study groups in the judgement-free comfort of peer-to-peer 
support. Enthusiasm for these programs was complemented by growing evidence of success in 
reducing course failures (Zaritsky 2006). Unlike tutorials where the Teaching Assistant’s (TA’s) 
instruction and related grading protocols frame the experience, SI is non-evaluative and peer- 
focused. Impact data shows overwhelming success in improving GPA performance in difficult or 
high-attrition courses (Zaritsky 2006). Terrion and Dauost point out that FSG can be especially 
helpful for first-year students as their persistence in school can be directly linked to engagement 
with faculty and peers: the more interactions that first-year students have, the more likely they are 
to remain enrolled in university (p. 312). In designing effective SI programs, educational 
developers, faculty, and staff continue to seek evidence-based guidance to implement high-impact 
programming. While peer-led FSGs on college and university campuses have historically been a 
proven strategy for retention ensuring university students stay in school, we knew from anectodal 
evidence on our campus SI impact goes beyond the “GPA boost” metric as students and FSGL 
leaders describe unique and lasting impact of the socio-academic network they build, the work- 
integrated learning they experience and transferable leadership competencies they adopt. As a 
result, UTM is now turning its attention to affective qualities and graduate outcomes starting with 
a focus ons FSG leaders, to assess to what degree students leading our program experience feel 
changed and transformed through the experience of leading instruction. 
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Inadvertently, the transition of the FSG program to a virtual environment and hyflex delivery 
model provides an opportunity to rethink the goals of the SI program more generally around how 
it might evaluated. We were initially guided in our thinking by Universal Design for Learning 
(UDL) a pedagogical framework we had employed in the transition to hyflex to help us think about 
and promote inclusive and equitable environments for all learners (Lee 2021; Ahhyun 2021). We 
were excited by the possibility of offering learning alternatives that would increase the number of 
SI options available to students using various “on-demand” practices We saw this as a positive 
first step to increase opportunity and access. The conceptual framework of UDL which advocates 
for the removal of barriers so that all students can experience meaningful engagement in their 
learning environment by incorporating multiple means for students to (1) engage with content, (2) 
represent content, and (3) express skills and knowledge. UDL frameworks help sus to see that not 
all engagement platforms will be optimal for all learners. We do know that SI programs, separate 
from the classroom experience in a credit course, already offer alternatives and are rooted in 
multiple representations. (CAST 2018). Tobin and Behling further highlight the intentionality that 
students bring to the learning experience when choosing to participate in SI, and how the openness 
and co-construction of knowledge can increase the personal connection students have to the 
material (p. 92). 


Initially only offered in-person, the urgent and immediate shift to remote delivery pushed FSGs 
through several iterations across different platforms. We began by testing FSG delivery on the 
learning management system BbCollaborate embedded into the university’s preferred platform 
Canvas. Our program managers and developers were especially concerned about data protection 
and privacy management, and these were the safest options at that time. Limited by some of the 
tools and functionality, we transitioned after term one line and test SI delivery on the platform 
Zoom as their safety and functionality became a more convincing option. In piloting these 
different technologies with our FSG Leaders, we were able learn processes for training and 
supporting FSGLs within this new learning space. This involved focusing on leveraging how the 
distinct features of oftware enabled virtual teaching pedagogies and ed tech could be used. While 
we saw UDL as a useful framework to expand our ideas about alternatives, we realized we had 
limited ways to understand how leaders were responding to uncertainty, volatility and complexity 
of the times. 


We became curious to know more about how our leaders were feeling stepping into these new 
roles of responsibly, guiding their FSG participants through uncertainty while modelling flexibility 
and self-efficacy. We noted that FSG leaders involved in this experienced enhanced engagement 
as we watched them respond, invest and restructure their programs to meet student learning needs. 
In noting the outcomes of these experiments in hyflex learning, we understood immediately that 
prioritizing PsyCap as an outcome of FSG training should be prioritied. As Debebe outlines, 
positive PsyCap is a systematic process for developing learned resourcefulness. In this process, 
the instructor intentionally cultivates hope, self-efficacy, resilience, and optimism in the student 
(Debebe 2017) to eliminate deficit thinking and, in turn, intervene with ascription. Ascriptions are 
the beliefs that students hold about themselves and their potential, as cultivated through their socio- 
environmental experiences. As we saw FSGLs succeed and adapt to new technologies, applying 
the model of PsyCap would shape our goals and aims for leader training and alertus to specific 
“look-fors in professional development.” Consistent with Lozada (2017), our experiment 
prioritized a better understanding of how “SI facilitators experience transformative learning and 
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the nature of civic engagement within their student leadership roles” (p. 80). Data shows many of 
our FSGLs leave university and continue to significantly contribute within their community and 
professional roles post-graduation. We hope to learn more about the impact of the FSG program 
on those talent trajectories and, in turn, expand our initial Universal Design for Learning (Lee 
2021; Ahhyun 2021) framework accordingly. 


A Framework for Leadership Engagement 


The evolution of the FSG program at UTM is indebted to benefactor Robert Gillespie, who in 
donating funds to help found SI on our campus had this to say of his university experience: “I felt 
too much emphasis was placed on learning by rote and not enough on reaching conclusions based 
on deduction, experience and collaborating with others.” With this in mind, our specialized 12- 
week undergraduate course EDS325: Supplemental Instruction in Higher Education was created 
to extend collaboration opportunities for FSG leaders in a course that has been successful in 
attracting highly motivated FSGLs across the university’s disciplinary programs. The course has 
become an important training ground for high performance as learning outcomes were shaped to 
match and promoting PsyCap outcomes. FSGLs participate in self-advancement as they learn 
about themselves, their purpose, and the unique contributions they can make to the broader 
community while practising and applying skills of professional facilitation, public speaking and 
learning about theory and research around group dynamic, engage in equity challenges and critical 
thinking as they put into practise the experience in the use of educational technology. Operated 
in collaboration with the Program Manager and taught by two professors in the Education Studies 
program, student leaders taking the course practice crisis management and participate in theory, 
research, and in the application of case studies through problem-based learning. In this hyflex 
course, there is a focus not only on the obvious inventory of virtual pedagogies (e.g., using 
breakout rooms, engaging with the chat) but on the experience and outcomes of the FSGL through 
the lens of reflective learning. 


Mezirow’s pioneering work in the field of transformational learning found that the integration of 
critical reflection and peer review could lead to a transformation in the way an adult learner 
constructed their understanding of knowledge (1991). Within this theory, frames of reference that 
are constructed through previous experiences are challenged each time individuals encounter new 
experiences. When these new interactions question our established frames of reference to the 
point of questioning the presuppositions on which the references are based, then the intervention 
could lead to a revision of the initial frame of reference, thus transforming them. Mezirow’s 
theory has been used to substantiate the use of critical reflection in a variety of learning 
environments, from the development of the Scholarship of Teaching and Learning (Kreber 2006) 
to how transformation can be used to enhance other learning theories, such as threshold concepts 
(Hodge 2019). 


In pivoting to a virtual format, we were initially excited by the many alternatives that we could 
offer our FSGLs. We continued to pursue, however, what this shift to remote and now hyflex 
learning might mean for student engagement. Enhanced engagement is a view of academic success 
that goes beyond what needs to be learned and focuses on how we learn, taking into account 
identity, learning processes, and community needs. It is centered on a co-constructivist approach 
that involves promoting opportunities for students to become active learners through a process of 
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civic involvement that focuses on the transformation that takes place from those experiences. 
Enhanced engagement fosters a sense of collective and individual identity-building where students 
see their role as participants in the collaborative reconstruction of knowledge. 


Indeed, the type of student engagement achieved through FSGs is of particular interest to us in this 
current study as we hade witnessed many of the same benefits reported in recent studies on student 
engagement (collated in Dawson et. al. 2014), including enhanced social relationships through 
student-to-student contact (Mahdi 2006) as well as a reduction of anxiety towards difficult courses 
due to the supportive environment and increased opportunities to talk through challenging course 
material with others (Bronstein 2008). Additionally, we identified a connection between agency 
and success, wherein the voluntary aspect of FSGs afforded students a sense of control over their 
learning experience, resulting in a small but significant contribution to success in post-secondary 
education (Richardson et al. 2012). 


Explorations in Virtual, In-Person, Hyflex Delivery 


AS we experimented with online platforms and technologies to create new learning environments 
for our students and FSGLs, we discovered that our own evolution of platform and approach 
supported and nurtured resilience in our leaders. To understand where we might go next in FSG, 
we thought it important to first look at the past. Innovation in SI has been going through a slow- 
paced but constant evolution since its inception, moving from in-person learning among small 
groups, shifting to flipped classroom experiences in some schools, and now arriving at a hybrid 
configuration of interactions on college and university campuses. As Alden (2017) points out, 
recent innovations in SI have come naturally alongside the proliferation of social media and advent 
of online learning complements that enhance what traditional SI provides. She explains, 
“Innovations include (but are not limited to): video lectures, practice exams, and promotion of 
group collaboration among students. Through the use of platforms including Facebook, Twitter, 
Piapp, and Slack, the SI session can be held almost exclusively online and provide just as much if 
not more benefit to the students involved” (p. 2007). As new options open up further alternatives 
for SI, we can see how a targeted preparation of FSGLs would be essential to the effective, 
sustainable delivery of FSGs within a virtual environment, and that establishing such a foundation 
now would ensure the success of the program in the years to come. 


The University of Toronto Mississauga “virtual cineplex” FSG platform, named by its designer as 
a metaphor for the multiplicity of options that one can choose from in this online, on-demand 
space, provides a portal by which students could easily enter and join FSG classes offered around 
the clock in a multitude of disciplines. Providing alternate formats to engage with FSGs, including 
an accessible virtual format, created opportunities for a much wider population of students, 
bypassing the commuting delays that often impact in-person scheduling. At the same time these 
new integrations presented challenged to our leaders to adopt to new ways of doing things. For 
example, initial data indicated that allowing cameras to be turned off during FSGs was helpful for 
students who wished to remain anonymous on Zoom and “listen in” on conversations. However, 
what FSGL’s deemed an inclusive practice, it turns out, significantly reduced levels of meaningful 
engagement as visual communication cues were eliminated and networking and community 
building opportunities were minimized. Leaders had to find other ways to achieve the same results 
as they had experienced during in-person FSGs,. These setbacks led to ongoing testing of different 


362 


and new virtual pedagogies and, by default, created work-integrated real-world simulations of 
problems that our leaders had to solve. With findings from each new prototype of FSG 
development, leaders understood that new technology in SI was never going to fix all pedagogical 
problems. They readjusted their expectations and set out to understand and reflect on practice as 
they learned more. We saw in this how digitization and new tools could be adopted with the 
intention to enhance our aims and amplify opportunities for our leaders to work in a safe trial and 
error environment where they could collaborate and reflect and, in turn, build PysCap as 
transferrable skills future problem solving. Despite the benefits of VFSGs, and this resulted, in 
some cases, to more inconsistent attendance than among in-person iterations of FSGs. 
Simultaneously, however, the additional skills developed in VFSGs, including mastery of 
technology, helped us to reconfigure ideas about how collaboration occurs online and to define 
essential competencies we wanted to share with our virtual FSGLs. On the other hand, many 
students reported increased engagement in their VFSGs due to the multiple modalities of 
engagement, including texting via the chat feature. The availability of these features allowed 
students to participate more freely and suggest answers more readily, and with greater confidence, 
regardless of their certainty towards their answers. Multiple studies suggest that greater, rather 
than lesser, confidence in an incorrect answer will increase the likelihood of remembering the right 
answer when it is corrected (Metcalfe 2017). The VFSGs provided enough instances of greater 
access to our SI offerings, so as to support our pursuit of new visions of student engagement in 
future SI iterations. 


Enhanced Engagement Opportunities 


We quickly began to see FSGLs as our co-constructors in hyflex remodelling. With that, a new 
“constellation of opportunity” (see Figure 1.1) has emerged in our program. In the process, we 
saw a clear path for nurturing programs that focus on what Portelli describes as the “curriculum of 
life” (p. 59). For the past decade or more, McMahon and Portelli have asked questions about what 
student engagement looks like and feels like, examining and interrupting popular discourses on 
this topic. Consistent with their view of engagement, we see opportunities to address not only the 
technical needs of our new virtual environment, but the need to foster transformative learning 
experiences. We broke from training approaches “simply as a matter of techniques, strategies or 
behaviours” and instead built into the program more “intrinsically the purpose of democratic 
transformation where everyone has an opportunity to bring their needs to the table and participate 
in understanding the curriculum of their lives” (McMahon & Portelli p. 70). This view of the FSG 
program allows us to expand from a minimalist or homogentisic perspective where student 
engagement is simplified down to a “correlation between engagement and academic achievement” 
(Finn & Voelkl 1993; Newmann, Wehlage & Lamborn 1992; Steinberg 1996, from Portelli 2004). 
Instead, we see the FSG environment as one in which each stakeholder, from the EDS325 course 
instructor to FSGLs and Program Assistants, is involved in unwrapping, unwinding, and 

reconstructing knowledge together. FSGs evolve according to what each ndividual brings to 
inform individualized mentoring, and related competency development in our leaders. Figure 1 
shows the new constellation of opportunity we provide for leader development Step 1 involves 
access to pariticipation in FSG sessions. Step two involves providing opportunities for those 
students who have taken the FSG to then sign up for the hyflex EDS 325 course and experience 
training sessions on leadership and PsyCap topics. Leaders can then move into the teaching 
assistant role in a paid position within the EDS 325 Supplement Instruction course and/or become 
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an RA and program assistant in the program. As a result of program assistantship, they can be 
involved in research in SI and receive one-on-one mentoring to support post-graduate pathways. 
We know that students enrolled in the same course have the opportunity to study together in peer 
clusters, the process facilitates the development of academic skills (Hurley & Gilbert 2008). The 
weekly FSG leader course guides the group through problems and issues that participants identify 
need focus. WE continue to want to know more about the potential lasting impact these learning 
communities for our FSGLs have on talent trajectories as FSGLs learn alongside their peers in an 
active learning setting that utilizes team-based learning (TBL; Silva et al. 2021) 


Figure 1.1: Constellation of Opportunity in Supplemental Instruction 
University of Toronto Mississauga 


Figure 1.1: Constellation of Opportunity for FSG Leader Development 
University of Toronto Mississauga Supplemental Instruction Program 
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Cultivating Psychological Capital (PsyCap) in a VFSG Environment 


As Lozada (2017) proposes, “Providing students an opportunity to serve in a leadership role can 
lead to the actualization of transformative learning experiences, which may materialize in a 
heightened development of skills that are transferable to future academic, professional, and civic 
aspirations” (iv). Building a community for FSG leaders has really been a response to students 
needs through community. We are cognizant that cooperative learning goes beyond physical 
proximity or the mere sharing of information in a group setting, and that it needs to be structured 
in order to be effective (Johnson & Johnson 2002). In the virtual learning environment, the 
importance of structuring cooperative activities is heightened due to the increased sense of 
isolation students feel when they are not sharing a physical space. Students and instructors have 
reported feeling “disconnected” in virtual classroom environments, and offering opportunities for 
true collaboration can help to mitigate these feelings. To address this, we instructed FSGLs on the 
five basic elements of cooperative learning: (1) positive interdependence, (2) individual 
accountability, (3) face-to-face promotive interaction, (4) social skills, and (5) group processing 
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(Johnson & Johnson 1989, 1999a; Johnson, Johnson & Holubec 1998a). The elements we found 
to be particularly helpful in the VFSG environment for leaders were positive interdependence and 
face-to-face promotive interaction. Positive interdependence relies on the perception that students’ 
success is inextricably linked to the success of others: individuals succeed when the group 
succeeds and vice-versa (Johnson & Johnson 2002). In our VFSG environments, for example, we 
promoted activities in which each group member was assigned a portion of an overall assignment, 
as in the jigsaw cooperative learning strategy. By coming together to share and discuss their 
findings, students discover the completeness of the activity. Learning is “complete” only when the 
group comes together and all of the individual contributions make up the whole. 


Face-to-face promotive interaction involves encouraging the collective as they complete their 
tasks, as well as facilitating the completion of the tasks. Both are employed in order to reach the 
group’s goals (Johnson & Johnson 2002). The virtual iteration of this cooperative element is 
fundamental to establishing and maintaining not only group morale but group cohesiveness as 
they work on cooperative tasks together. We found that frequent and targeted promotive 
interactions helped the students stay-connected. Other techniques such as committee-building, 
lesson study, breakout rooms, micro-teaching, and guest speakers have all become key parts of 
our FSGLs’ learning experience. 


CONCLUSION 


At the University of Toronto Mississauga, we are presently in an important trial-and-error phase 
of program-level innovation in SI. This was the result of the hard pivot to VFSG and the new for- 
credit academic course that we adopted as part of SI instruction. We know that the FSG program 
on our campus attracts students who are already on a leadership path and the offerings we provide 
are designed to amplify the very qualities that attract students to the program. 


George Couros, in his book Innovate Inside the Box, explains that reflection is a key component 
of any kind of process. Incorporating learning from past mistakes is an essential step in innovation 
(Couros 2019). In this paper, we have responded to Couros’ suggestion by reflecting on what the 
future of enhanced engagement for SI leaders might look like while identifying challenges and 
constraints that we encountered while responding to new opportunities in hyflex delivery. 


We wanted to know how FSG leader training is enhanced or limited by the use of virtual 
technologies and whether barrier reductions could be achieved by a virtual SI program that utilized 
multimodal approaches. We found that this innovation indeed augmented opportunity. Using a 
UDL framework to guide development, we applied criteria to assess alternatives in our programs, 
which helped us to recognize the need to better plan for learner variability in our student 
population. As David Gordon suggests, we can do this by providing students with options: 


Options are essential to learning, because no single way of presenting information, no 
single way of responding to information, and no single way of engaging students will work 
across the diversity of students that populate our classrooms. Alternatives reduce barriers 
to learning for students with disabilities while enhancing learning opportunities for 
everyone (Council for Exceptional Learning, 2011 in Tobin and Behling, 25). 
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This virtual iteration of our SI programs inadvertently provided opportunities for multiple means 
of engagement, representation, action, and expression—namely, the main criteria of the CAST 
UDL guidelines (CAST 2018). Through this methodology, we continue to seek to uncover the 
degree to which these options have been beneficial and to provide a framework for how to 
intentionally incorporate enhancements into our SI FSG Leader training programs, thereby 
fostering more inclusive alternatives for all students. 


The introduction of a new hyflex credit-bearing course for FSGLs in leadership and self- 
development added a significant opportunity for enhanced engagement. One focus that was 
important to us was to ensure that our FSGLs and Program Assistants were provided with adequate 
training in both VFSGs and in-person FSGs by utilizing the for-credit course to not only teach 
leadership theory and scholarship but to centre time in that class on strategic life planning that 
widen opportunities for civic engagement and the exploration of post-graduate pathways. 
Responding to the needs of our student FSGLs, many of whom apply to professional post-graduate 
programs in areas such as medicine, nursing, teaching, and law, saw the relevance in FSG 
instruction and virtual with the aim to focus learning outcomes and mentoring on affective domains 
by creating an environment both online and inperson that cultivates of PysCap 


Thinking beyond the borders of both the classroom and the institution, a next step in understanding 
FSG leader experiences is to propose a longitudinal study to follow FSGLs post-graduation to 
understand the transfer of qualities and characteristics after university. Rather than focusing on the 
metric of GPA improvement in the student groups they serve, we propose using the four pillars of 
the PsyCap metric (HERO: Hope, Efficacy, Resilience Optimism) to evaluate trasferable qualities 
of leadership that our leader take with them in the characteristics associated with fulfillment, 
wellbeing, and thriving. The longitudinal study of these newly proposed post-graduate outcomes 
will allow us to better assess our unique FSG train the trainer program and more fully evaluate ithe 
potential of a 12 week course and hyflex alternatives as a means to foster transformational for FSG 
leaders. 
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Abstract 

Research has established both the importance of integrating technology in English language 
learning and the importance of properly training teachers in order to integrate technology 
effectively. While this is true for English as a second language teachers in all contexts, there is 
a strong need for instructors working with adult learners to utilize technology and be properly 
trained in technology integration. The integration of professional learning communities or 
communities of practice and coaching/mentoring been shown to be effective professional 
development interventions. For professional development to be successful, it should cater to the 
specific needs of the instructors; it is therefore important to understand the needs of the 
instructors and to explore the different types of professional development that fit their needs. 
This research explored the needs, current technology integration practices, and attitudes toward 
the use of technology with English to Speakers of Other Language (ESOL) educators within a 
community college context in order to make recommendations for professional development in 
technology integration. Using an interpretive-descriptive qualitative design, this study utilized 
classroom observations, one-on-one instructor interviews, a focus group interview, and a 
researcher journal. Acting as a needs analysis, these sources were analyzed inductively in order 
to make professional development recommendations. Based on the needs of the study’s 
participants, it was determined that they would benefit from a technology mentor/coach and a 
professional learning community or community of practice to provide support as well as offer 
collaboration opportunities, resulting in increased technology integration including both 
instructor and students uses of technology. 

Keywords: professional development, technology integration, ESOL instructors, 
interpretive-descriptive qualitative design, community college 
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Research has established both the importance of integrating technology in English 
language learning and of properly training teachers to effectively integrate technology into their 
teaching practices (Kessler, 2018). While this is true for English as a second language (ESOL) 
teachers in all contexts, there is a strong need for instructors working with adult learners to 
utilize technology and be properly trained in technology integration (Bergey et al., 2018). 
Professional development (PD) is a way to increase successful technology integration and to 
overcome barriers that prevent it. For PD to be advantageous, it should cater to the specific needs 
of the instructors within the context of their instruction (Kopcha, 2012; Oliver & Townsend, 
2013). This research explored the needs, current technology integration practices, and attitudes 
toward the use of technology of four ESOL educators within a northwestern United States 
community college context in order to make recommendations for PD in technology integration. 

Literature Review 

Integrating technology into language teaching has shown to aid learners in reaching 
higher proficiency levels, as well as increase their autonomy (Adair-Hauck et al., 2000; Healey 
et al., 2011). Healey et al. (2011) identified three general themes that target a need for 
technology standards for the field of Teaching English to Speakers of Other Languages 
(TESOL). Those themes include: (A) research shows that there are benefits from the use of 
technology in language learning and teaching; (B) technology should be integrated to support 
acquiring the second language and to develop electronic literacy; and (C) research shows that 
technology in learning is not being used to its fullest potential because of inadequate teacher and 
learner training. The international TESOL organization developed these themes within its 
framework for best practices of technology integration in the language classroom because of the 
positive impact it has on English language teaching/learning (Healey et al., 2011). The benefits 
technology has for teaching adult English language learners in the U.S. include increasing 
motivation, engaging students in learning, providing authentic language use, and accommodating 
diverse learners (McClanahan, 2014). Eyring (2014) suggests that technology is a valuable tool 
for adult English language learners and can be transformational in increasing literacy, engaging 
students, increasing proficiency, and exposing learners to 21°‘ century skills needed in the 
modern world. While it is recognized that there are challenges and barriers to using technology 
within adult ESOL programs (McClanahan, 2014), the benefits are too great not to overcome the 
barriers. The use of technology in language teaching and learning has been established as 
beneficial enough to have a dedicated subfield, computer-assisted language learning (CALL). 
Even though CALL is necessary to successfully utilize it for language teaching and learning, the 
majority of training is acquired through conference workshops, personal reading, and other 
modes of self-education (Healey et al., 2011; Kessler, 2006, 2007). 

Teacher training and professional development (PD) regarding technology integration in 
English language teaching/learning needs to take place in order for technology to be utilized 
effectively (Arnold & Ducate 2015; Healey et al., 2011). Chapelle (2008) recognized that even 
though some language professionals have had limited technology training, such as in a general 
education course, more training is necessary to understand specifics about technology in 
language teaching. While this is true for ESL teachers in all contexts, there is a strong need for 
instructors working with adult learners to utilize technology and be properly trained in 
technology integration (Chisman, 2008; Warschauer & Liaw, 2010). 

Three large bodies of research established both the importance of integrating technology 
with adult English language learners and the importance of properly training teachers to integrate 
technology: (A) The Center for Applied Linguistics (2010); (B) Passing the Torch: Strategies for 
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Innovation in Community College ESL (Chisman & Crandall, 2007); and (C) TESOL 
Technology Standards (Healey et al., 2011). The Center for Applied Linguistics (2010) 
determined that practitioners working with adult English language learners need continual PD 
following a specific framework, which they developed because of the rapid growth of the 
immigrant population in the United States in the last 20 years. In this framework, one of the 
essential elements is the appropriate use of technology to support learners before, during, and 
after their courses. Chisman and Crandall (2007) conducted one of the largest studies regarding 
adult ESL community college programs in the United States, which studied five exemplary ESL 
community college programs for two years. The study revealed that a contributing factor to the 
success of these colleges were a variety of PD activities offered through the colleges to their 
faculty and staff, including ongoing technology training and support (Chisman & Crandall, 
2007). The study found that in-house PD and support is “essential to maintaining a high-quality 
faculty” (p. 91). In the TESOL technology standards, it is stated that there is a lack of proper 
training among ESL teachers and learners regarding effective uses of technology in English 
language learning (Healey et al., 2011). This suggests that PD is necessary in order to support 
teachers in technology integration in the language classroom. 

PD provides the opportunity for teachers to acquire new perspectives, knowledge, and 
skills through both formal and informal experiences; these experiences come in a variety of 
formats including structured in-service trainings, peer teaching, mentoring, books clubs, and 
informal discussions (Coldwell, 2017; Gaines et al., 2019). PD is considered effective when 
teacher practices are improved and student achievement increases as a result (Evens et al., 2018; 
Twining et al., 2013). In order to determine what type of PD will best support teachers within a 
given context, it is important to understand the needs of the instructors within that context. 
Oliver and Townsend (2013) and Kopcha (2012) assert that a needs assessment is important in 
developing PD opportunities that cater to the needs of a specific teacher population and their 
context. A needs assessment is an effective way of determining the internal/personal factors, 
such as beliefs, that teachers hold that may impact the type of training that is best for them 
(Kopcha, 2010; Vatanartiran & Karadeniz, 2015). Therefore, prior to implementing PD in 
technology integration, it is essential to fully understand the context in which it will take place 
(Kopcha, 2010, 2012; Oliver & Townsend, 2013). Additionally, in developing and implementing 
PD opportunities for educators, it is important to align with theories of adult learning, which 
emphasizes the self-directed nature of these learners (Beavers, 2009; Center for Applied 
Linguistics, 2010). 

Using PD as a vehicle for increasing technology integration practices is supported by 
numerous models for effective technology integration. To understand best pedagogical 
application for technology integration in classroom practices, the Technological Pedagogical 
Content Knowledge framework (Mishra & Koehler, 2006), the Technology Acceptance Model 
(Davis, 1989), and the Diffusion of Innovations theory (Rogers, 2003) were appraised. These 
models agree that PD in technology integration can lead to more effective technology integration 
in the classroom and suggest that ongoing PD is necessary for teachers to increase and improve 
their technology integration practices. In order to determine how PD should be designed to meet 
their needs and aid in overcoming common barriers found in the adoption of new technologies, 
understanding the pedagogical beliefs and self-efficacy values of the teachers involved, as well 
as the environment in which they teach, including the characteristics of their learners and their 
institutional context, is crucial. 

The context of this research was in an adult ESOL program at a Pacific Northwest 
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Community College (PNWCC, a pseudonym) in Oregon. ESOL programs within community 
colleges fall within adult basic skills and are committed to aiding their learners in meeting the 
adult learning standards that align with the National College and Career Readiness Standards 
(Oregon Higher Education Coordinating Commission, n.d.). The standards for English language 
arts and literacy include several domains including, reading, writing, speaking, and listening. 
Within the standards across these domains is the analysis and integration of information from 
media to reflect the importance of the students’ ability to adapt and utilize new technologies 
(Pimentel, 2013). The adult learning standards recognize the importance of technology in 
teaching and learning to prepare learners for the skills they need for work or educational 
endeavors after the community college. 

Methodology 

The lead researcher of this 10-week study was immersed in the ESOL community college 
culture as a peer instructor. Because of the exploratory nature of the study, action research with 
an interpretive-descriptive qualitative design was used to systematically explore and analyze a 
phenomenon that allowed for recommendations for future action (Thorne, 2016). As described 
by Thorne (2016), interpretive-descriptive qualitative research was brought into existence 
through observation of the features and characteristics that make for valuable qualitative studies 
for real world application. Through classroom observations, one-on-one interviews, a focus 
group interview, and a researcher journal, the participants in this study illustrated their 
experiences with technology in their teaching, as well as the barriers they face in using it more. 
In exploring the needs, technology integration practices, and attitudes toward technology of the 
ESOL instructors at PNWCC, ideas and recommendations for PD regarding how to aid 
instructors in increasing their technology integration practices emerged; therefore, this study 
acted as a needs analysis for a type of instruction. 

Setting and Participants 

The setting for this study took place within the ESOL department at PNWCC, which 
comprised five faculty, serving approximately 125 students. Students in this program receive 
English language instruction in the four major language skill areas of reading, writing, speaking, 
and listening. There is not an established curriculum in place for this program. There are, 
however, learning standards and benchmarks, all related to language skills needed in real life 
situations, which establish the normative standard within the local context. Instructors in the 
program are allowed to choose what standards and benchmarks they want to include in their 
courses. Benchmarks are the focus because they are intended to act as objectives, containing 
information about the instructional focus, and identifying skills students will need to practice 
(Oregon Office of Community College and Workforce Development, 2017). Embedded within a 
large portion of these benchmarks are expectations for students to utilize technology to support 
the development of their language and computer literacy skills. 

Courses in the ESOL program at PNWCC are provided for students of four different 
proficiency levels: (A) Beginning Literacy/Low Beginning, (B) High Beginning, (C) Low 
Intermediate, (D) High Intermediate/Advanced. Each course meets twice a week for three hours, 
for a total of six instructional hours a week. The program consists of non-credit classes and 
students are able to stay in the program for as long as they like. Classes are taught in modern 
classrooms equipped with a podium that has a computer, internet access, sound system, 
projector, and document camera. Some classrooms have circular tables for students and some 
have longer tables, where students sit side by side. No classroom has individual desks. For one 
hour a week, each course is expected to meet in a computer lab on campus, where all students 
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have access to their own computers. Four PNWCC ESOL department instructors agreed to 
participate in this study. The participants range in age from 40-60, three have Masters degrees in 
TESOL and one instructor has a Master’s in Education degree. 

Data Collection 

Data was collected in the form of classroom observations using a modified version of the 
Looking for Technology Integration (LoFTI) instrument (William & Ida Friday Institute for 
Educational Innovation, 2010), one-on-one instructor interviews, a focus group interview, and a 
researcher journal. The purpose of the LOFTI instrument is to aid in the observation of 
technology integration into teaching and learning. The data gathered through the use of this 
instrument is helpful in planning and/or providing professional development in instructional 
technology (William & Ida Friday Institute for Educational Innovation, 2010). Classroom 
observations added to the conceptualization of the technology integration practices of the 
instructors. Each instructor, who participated in the study, was observed once for 45-50 minutes 
of a three-hour class. It was requested that the instructors use technology in some capacity during 
the portion of the lesson observed. As included in the LoFTI tool, PNWCC teacher activities 
with technology were observed, such as the use of technology to activate prior knowledge, 
differentiate instruction, lecture, and summarize. The use of technology for assessment was also 
observed. 

One-on-one instructor interview questions offered an understanding of the participants’ 
technology integration practices and attitudes toward technology, as well as their needs and the 
barriers they face. These interviews were face-to-face, semi-structured, and lasted approximately 
20-30 minutes. Base questions for the interview were in place with follow-up questions asked as 
needed (Mertler, 2017). Gathering this information was important to determine how to design 
and implement an effective PD that may result in increased technology integration practices. 

The focus group interview provided a more in-depth conception of the instructors’ 
technology integration practices, but especially the barriers they face. The focus group interview 
also offered suggestions and ideas from the instructors about how to overcome these challenges 
and increase their technology integration practices. The focus group interview included open- 
ended interview questions and occurred after the classroom observations and instructor 
interviews were conducted. 

A researcher journal was kept during the entire data collection process to document and 
recap interviews, email exchanges, and observations that took place at various other points 
during the data collection process. The lead researcher’s reflective perspective offered in the 
journal provided a place to write about what happened immediately following the event, as 
opposed to relying on memory. 

Data Analysis 

Acting as a needs analysis, these qualitative data sources were analyzed inductively in 
order to make recommendations, in collaboration with the ESOL faculty, regarding professional 
development in technology integration. The total number of digitized data sources uploaded to 
Delve software for coding was 13. From these 13 sources of data, 1,371 codes were applied. 
From these codes, Saldafia’s (2016) first and second cycles of coding took place resulting in 10 
final categories and three themes materialized. These three themes supported the assertion: 
Participants discern that the attributes of technology use outweigh student and instructor barriers 
for English language teaching and learning within this context (see Figure 1). 
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Figure I Assertion with supporting themes and categories. 


In this study, the findings suggested that PD in technology integration within the 
community college ESOL be specific to the unique needs of the instructors. These findings were 
congruent with existing literature regarding PD in technology integration, PD in CALL, and PD 
within the ESOL community college context. Ottenbreit-Leftwich et al. (2010) and Kopcha 
(2012) both emphasize situating PD in technology integration to address the needs of the 
teachers that are specific to their environments. Kopcha (2012) suggests that situating 
professional development can aid in overcoming barriers such as vision and beliefs. The 
contextualized nature of CALL training is also emphasized throughout research, where it 
encourages that professional development focus on technologies that are applicable to the 
context of focus (Almuhammadi, 2017; DelliCarpini, 2012; El Shaban & Egbert, 2018). 
Situating and contextualizing PD based on the unique characteristics of the context is also 
recommended for PD for community college ESOL instructors (Rodriquez & McKay, 2010). 
Young and Petyon (2008) recognize the complexities of designing PD opportunities for 
educators working with adult ESOL learners in community colleges, and recommend using a 
data-driven, systematic process to determine the needs of these practitioners in order to plan for 
PD. 

This study acted as a data-driven and systematic approach in determining the needs of the 
PNWCC participants. Without fully exploring their needs and coming to understand their 
barriers based on their experiences, it would not have been possible to recommend avenues of 
PD that met those needs. The input from the participants regarding their experiences and their 
ideas guided the recommendations, aligning with theories of adult learning (Chen, 2014), which 
encourage participants to be involved in decisions about PD. Trotter (2006) claims that teachers 
should be given freedom to develop PD opportunities based on their needs and personal interest. 
The recommendations made as a result of the findings of this study offered the participants the 
opportunity to determine the direction of the PD. 
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Recommendations for Professional Development in Technology Integration for ESOL 
Instructors 

It is hoped that the TESOL technology standards will motivate professional organizations, 
teacher education departments, and individual English language programs to evaluate and 
educate their teachers to meet targets articulated in the performance indicators of the 
technology standards (Healey et al., 2011). As has been established through the existing 
research and as evidenced in these findings, technology is a valuable tool for teaching and 
learning within this context. Participants recognized the benefits technology has for 
instructional purposes and utilized it frequently. Their uses of technology were largely driven 
by the needs and interests of their learners, as were the types of technological activities and 
resources used in their classes. These activities were centered on preparation for language use 
in the real world and were supported through the use of authentic materials, which were 
identified having been found through the internet. Participants realized that they could be using 
technology more, especially with their learners, and presented ideas for increasing practices in 
that regard. They had also identified PD opportunities specific to their needs within the 
situation of their context. The following PD for increased technology integration was 
identified: 

1. A technology lead who: 

a. Determines suitable websites for learners and creates a simple link or icon to 
these websites with an accompanying handout that lists these sites for 
learners to take home. 

b. Creates and organizes an online space for instructors to share resources and 
experiences. 

2. Collaboration in the online space, where resources, lesson plans, and experience are 
shared. 

3. Collaborative meetings twice a year to share ideas and collectively share resources 
and organize the online space to ensure continuity in instruction and resources for 
students. 

Technology Mentor/Coach 

Oliver and Townsend (2013) state that having a technology mentor or coach is a form of 
technology integration training, where those who are well-trained or experienced with 
technology support their less experienced colleagues. Peer coaching and mentoring provide 
collaboration and reflection, which are considered key components in effective PD (Sprott, 
2019). They lead to positive outcomes regarding the increased use of technology integration in 
classroom practices (Charbonneau-Gowdy et al., 2016; Oliver & Townsend, 2013; Richter et al., 
2011; Sprott, 2019). Mentoring or coaching provides an opportunity for expanding perspectives, 
analyzing preconceived notions, and sharing expertise to support adult development (Drago- 
Severson, 2008). The participants in this study identified the need for a technology lead from 
within the department who can vet suitable websites for learners, create and/or spearhead the 
creation of a simple URL or icon to these websites with an accompanying handout for students to 
take home, identify strategies for helping students increase their use of technology in the 
classroom, and develop and maintain an online repository for instructors to share resources and 
experiences, as well as facilitate collaborative meetings to further develop the online space. This 
technology lead could be considered a mentor or coach. 
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Professional Learning Community/Community of Practice 

Having a shared space could be considered a type of professional learning community 
(PLC) or community of practice (CoP), where there is a small group engaged in collaboration, 
discussions, the sharing of related resources, and a common practice (Jones et al., 2011). PLCs 
and CoPs are shown to increase technological knowledge and skills regarding technology in 
education (Cifuentes et al., 2011; Jones et al., 2011; Thoma et al., 2017). This can be attributed 
to the support, collaboration, and reflection offered through participating in a PLC or CoP. 
Research supports those opportunities for collaboration are among one of the characteristics 
that lead to successful PD (Bostancioglu, 2018; Sheffield et al., 2018; Sprott, 2019; 
Wennergren, 2015). PLCs and CoPs are forms of ongoing professional development that can 
better support educators than traditional forms of PD, such as one-shot workshops (Smaldino et 
al., 2012; Stewart, 2014; Thoma et al., 2017). Based on the idea generated by participants, a 
PLC/CoP is recommended for them. This would allow for continued communication via face- 
to-face meetings similar to how the focus group interview was structured. It was suggested that 
these collaboration meetings take place twice a year where sharing ideas, experiences, and 
resources could be a mechanism for overcoming challenges and increasing technology 
integration. 
Action Research 

As established by Dawson (2012), action research is a powerful vehicle for professional 
development, particularly within the realm of technology integration, as it can offer teachers an 
intentional study of the ways that technology impacts student learning, as well as “a lens through 
which teachers may experience conceptual change regarding their beliefs about technology 
integration practices” (p. 117). Rodriguez and McKay (2010) suggest action research is a 
particularly effective option for practitioners working with adult English language learners 
within programs in the U.S. because of the unique needs of the experienced teachers within this 
context. They also indicate that mentoring/coaching and peer observations could provide the 
opportunity for teachers to step out of their normal teaching roles and develop new paradigms for 
their work (Rodriguez & McKay, 2010). Because of the ability action research has to positively 
impact a change in teaching practices (Avalos, 2011; Dawson, 2012; Manfra & Bullock, 2014), 
another cycle of action research is recommended for PNWCC as a form of professional 
development. In addition to identifying a technology mentor/coach, creating a collaborative 
online space, forming a PLC/CoP, and meeting biannually, the integration of peer observation 
into the next cycle of action research is recommended. Peer observation offers a form of active 
learning and can play a role in successful professional development (Avalos, 2011; Richter et al., 
2011). 

Implications on Future Research 

The three themes and one assertion from the interpretive-descriptive qualitative analysis 
of this study offer implications for future research regarding PD for technology integration 
within the community college ESOL context. The interpretive-descriptive qualitative study, 
acting as a needs assessment, sought to fully understand the resources, skills, and concepts that 
the instructors within the proposed context currently had. Through exploring their interests, 
needs, insights, and ideas, the lead researcher and fellow faculty collaborated to determine PD 
endeavors that could be developed to best support them in increasing technology integration in 
their courses. This study could provide other researchers with a model for designing and 
conducting a study that acts as a needs assessment regarding technology integration within this 
context, or potentially within other similar contexts. It should also be noted that this study offers 
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a recent contribution to the body of research regarding effective PD opportunities for instructors 
within community college ESOL programs, where there seems to be a paucity in research. This 
study shows promise for utilizing a needs-based approach to designing technology integration 
PD for practitioners of adult community college ESOL through the use of an interpretive- 
descriptive qualitative design within a cyclical action research study. These findings could 
inform and guide others in developing a needs assessment within a context of focus in order to 
determine directions for PD. 

Limitations 
As is characteristic of qualitative research, limitations regarding the absence of quantitative data, 
ambiguities that inherently exist in human language, as well as the small population size need to 
be considered. Additionally, action research could be considered a limitation because it is 
focused solely on a problem identified within a specific context (Mertler, 2017), making it 
difficult to suggest the findings of this study as applicable to other contexts. A needs assessment 
conducted by an insider in collaboration with other insiders is a viable option to determining and 
creating PD opportunities. Further, the use of an interpretive-descriptive qualitative design 
within an AR model allowed for a thorough exploration of the participants’ situations based on 
their experiences that led to solutions to problems unique to them. While these limitations should 
be considered, they should not prevent others from using this study to guide and inform their 
own practices. 


378 


References 


Adair-Hauck, B., Willingham-MacLain, L., & Earnest-Youngs, B. (2000). Evaluating the 
integration of technology and second language learning. CALICO Journal, 17(2), 269- 
306. https://doi.org/10.11139/cj.17.2.269-306. 

Almuhammadi, A. (2017). EFL professional development: Discussion of effective 
models in literature. English Language Teaching, 10(6), 118-127. 
https://doi.org/10.5539/elt.v10n6p118. 

Arnold, N., & Ducate, L. (2015). Contextualized views of practices and competencies in 
CALL teacher education research. Language Learning & Technology, 19(1), 1-9. 

Avalos, B. (2011). Teacher professional development in teaching and teacher education 
over ten years. Teaching and Teacher Education, 27(1), 10—20. 

Beavers, A. (2009). Teachers as learners: Implications of adult education for professional 
development. Journal of College Teaching & Learning (TLC), 6(7). 

Bergey, R., Movit, M., Baird, A. S., & Faria, A. M. (2018). Serving English Language Learners 
in Higher Education: Unlocking the Potential. American Institutes for Research. 

Bostancioglu, A. (2018). Online communities of practice in the service of teachers' 
technology professional development: The case of webheads in action. Turkish 
Online Journal of Educational Technology-TOJET, 17(2), 97-110. 

Center for Applied Linguistics. (2010). Framework for quality professional development for 
practitioners working with adult English language learners. 
http://www.cal.org/caelanetwork/profdev/framework/. 

Chapelle, C. A. (2008). Computer assisted language learning. In B. Spolsky & F. M. Hult (Eds.), 
The handbook of educational linguistics (pp. 585-595). Blackwell Publishing Ltd. 

Charbonneau-Gowdy, P., Capredoni, R., Gonzalez, S., Jayo, M. J., & Raby, P. (2016). Brave 
forms of mentoring supported by technology in teacher education. Electronic Journal of 
e-Learning, 14(1), 3-14. 

Chen, J. C. (2014). Teaching nontraditional adult students: Adult learning theories in 
practice. Teaching in Higher Education, 19(4), 406-418. 

Chisman, F. (2008). Findings in ESL: A quick reference to findings of CAAL research onESL 
programs at community colleges. Council for Advancement of Adult Literacy. 
http://caalusa.org. 

Chisman, F.P., & Crandall J. (2007). Passing the torch: Strategies for innovation in community 
college ESL. Council for Advancement of Adult Literacy. http://caalusa.org. 

Cifuentes, L., Maxwell, G., & Bulu, S. (2011). Technology integration through professional 
learning community. Journal of Educational Computing Research, 44(1), 59-82. 

Coldwell, M. (2017). Exploring the influence of professional development on teacher 
careers: A path model approach. Teaching and Teacher Education, 61, 189-198. 

Davis, F. D. (1989). Perceived usefulness, perceived ease of use, and user acceptance of 
information technology. MIS Quarterly, 13(3), 319-340. https://doi.org/10.2307/24900 

Dawson, K. (2012). Using action research projects to examine teacher technology 
integration practices. Journal of Digital Learning in Teacher Education, 28(3), 
117-123. 

DelliCarpini, M. (2012). Action research: Building computer technology skills in teacher 
education. Language Learning & Technology, 16(2), 14-23. 
http://llt.msu.edu/issues/june20 12/action.pdf 


379 


Drago-Severson, E. (2008). Pillars for adult learning. Journal of Staff Development, 
29(4), 60-63. 

El Shaban, A., & Egbert, J. (2018). Diffusing education technology: A model for language 
teacher professional development in CALL. System, 78, 234-244. 

Evens, M., Elen, J., Larmuseau, C., & Depaepe, F. (2018). Promoting the development of teacher 
professional knowledge: Integrating content and pedagogy in teacher education. Teaching 
and Teacher Education, 75, 244-258. 

Eyring, J. L. (2014). Adult ESL Education in the US. CATESOL Journal, 26(1), 120-149. 

Gaines, R. E., Osman, D. J., Maddocks, D. L. S., Warner, J. R., Freeman, J. L., & 
Schallert, D. L. (2019). Teachers’ emotional experiences in professional development: 
Where they come from and what they can mean. Teaching and Teacher Education, 77, 
53-65. https://doi.org/10.1016/j.tate.2018.09.008 

Healey, D., Hanson-Smith, E., Hubbard, P., Loannou-Georgiou, S., Kessler, G., & Ware, 
P. (2011). TESOL technology standards: Description, implementation, 
integration. Teaching English to Speakers of Other Language, Inc. 

Jones, R., Fox, C., & Levin, D. (2011). Transforming education to ensure all students are 
successful in the 21st century. State Educational Technology Directors Association 
National Educational Technology Trends. 

Kessler, G. (2006). Assessing CALL teacher training: What are we doing and what could we do 
better. Teacher Education in CALL, 23-42. 

Kessler, G. (2007). Formal and informal CALL preparation and teacher attitude toward 
technology. Computer Assisted Language Learning, 20(2), 173-188. 

Kessler, G. (2018). Technology and the future of language teaching. Foreign Language 
Annals, 51(1), 205-218. 

Kopcha, T. J. (2010). A systems-based approach to technology integration using 
mentoring and communities of practice. Educational Technology Research and 
Development, 58(2), 175-190. https://doi.org/10.1007/s11423-008-9095-4 

Kopcha, T. J. (2012). Teachers’ perceptions of the barriers to technology integration and 
practices with technology under situated professional development. Computers 
and Education, 59(4),1109-1121. dio: 0.1016/j.compedu.2012.05.014 

Manfra, M. M., & Bullock, D. K. (2014). Action research for educational 
communications and technology. In Handbook of research on educational 
communications and technology (pp. 161-172). doi:10.1007/978-1-4614-3185-5_ 14 

Mertler, C. A. (2017). Action research: Improving schools and empowering educators. Sage 
Publications. 

Mishra, P., & Koehler, M. J. (2006). Technological pedagogical content knowledge: A 
framework for teacher knowledge. Teachers College Record, 108(6), 1017-1054. 

Oliver, K., & Townsend, L. (2013). Preparing teachers for technology integration: 

Programs, competencies, and factors from the literature. National Teacher Educational 
Journal, 6(3), 41-60. 

Oregon Office of Community College and Workforce Development. (2017). Oregon ABS 
learning standards. Prepared for the Oregon Adult Learning Standards Project. Salem, 
OR. 

Ottenbreit-Leftwich, A. T., Glazewski, K. D., Newby, T. J., & Ertmer, P. A. (2010). 

Teacher value beliefs associated with using technology: Addressing professional 
and student needs. Computers & Education, 55(3), 1321-1335. 


380 


Peyton, J. K., Burt, M., McKay, S., Schaetzel, K., Terrill, L., Young, S., & Nash, A. (2007). 
Professional development for practitioners working with adult English language learners 
with limited literacy. Proceedings of the second annual forum: Low-educated second 
language and literacy acquisition (pp. 213-225). The Literacy Institute at Virginia 
Commonwealth University. 

Pimentel, S. (2013). College and career readiness standards for adult education. Office of 
Vocational and Adult Education, US Department of Education. 

Richter, D., Kunter, M., Klusmann, U., Liidtke, O., & Baumert, J. (2011). Professional 
development across the teaching career: Teachers’ uptake of formal and informal 
learning opportunities. Teaching and Teacher Education, 27(1), 116-126. 
https://doi.org/10.1016/j.tate.2010.07.008 

Rodriguez, A. G., & McKay, S. (2010). Professional development for experienced teachers 
working with adult English language learners. CAELA NetworkBrief. 

Rogers, E. (2003). Diffusion of innovations. NY: The Free Press. 

Saldana, J. (2016). The coding manual for qualitative researchers (3rd ed.). Sage Publications. 

Sheffield, R., Blackley, S., & Moro, P. (2018). A professional learning model supporting 
teachers to integrate digital technologies. Issues in Educational Research, 28(2), 487-510. 

Smaldino, S., Lowther, D., & Russell, J. (2012). Instructional technology and media for 
learning (10" ed.). Pearson Education, Inc. 

Sprott, R. A. (2019). Factors that foster and deter advanced teachers’ professional 
development. Teaching and Teacher Education, 77, 321-331. 

Stewart, C. (2014). Transforming professional development to professional learning. 

Journal of Adult Education, 43(1), 28-33. 

The Oregon Higher Education Coordinating Commission. (n.d.). Adult basic skills 
resources. https://www.oregon.gov/highered/institutions-programs/ccwd/Pages/abs- 
resources.aspx 

Thoma, J., Hutchison, A., Johnson, D., Johnson, K., & Stromer, E. (2017). Planning for 
technology integration in a professional learning community. The Reading Teacher, 
71(2), 167-175. 

Thorne, S. (2016). Interpretive description: Qualitative research for applied practice. Routledge. 

Trotter, Y. D. (2006). Adult learning theories: Impacting professional development programs. 
Delta Kappa Gamma Bulletin, 72(2), 8-13. 

Twining, P., Raffaghelli, J., Albion, P., & Knezek, D. (2013). Moving education into the 
digital age: The contribution of teachers’ professional development. Journal of Computer 
Assisted Learning, 29(5), 426-437. https://doi.org/10.1111/jcal.12031. 

Vatanartiran, S., & Karadeniz, S. (2015). A needs analysis for technology integration 
plan: challenges and needs of teachers. Contemporary Educational Technology, 

6(3), 206-220. 

Warschauer, M., & Liaw, M. L. (2010). Emerging technologies in adult literacy and 
language education. National Institute for Literacy. 
http://eric.ed.gov/ERICWebPortal/recordDetail?accno=ED5 11970. 

Wennergren, A. C. (2016). Teachers as learners—with a little help from a critical friend. 
Educational Action Research, 24(2), 260-279. 

William & Ida Friday Institute for Educational Innovation. (2010). Looking for 
Technology Integration (LoFTI). N.C. State University College of Education. 


381 


Adults in Distance Education: A Multimodal Approach to 
Understanding Learner Engagement 


Anne Fensie, Jennifer Jain, Teri St. Pierre 
University of Maine 


Introduction 


In this paper, we propose a novel research methodology which used recorded study 
sessions with screencast, webcam video, and audio recording of adult students in their home 
environments as they participated in their online classes to understand the experience of adult 
learners in distance education. We will describe how we used screencast analysis to deconstruct 
study sessions to understand embodied actions distance students take during learning activities 
and the impact of their environment on their learning. This paper includes links to copies of the 
coding tool developed and sample video clips for analysis. A discussion of limitations and other 
applications of this research method are included. 


The State of Screen Recording Analysis 


Video analysis has a long history in the behavioral and education sciences (Erickson, 
2011), but the analysis of screen recordings has yet to become widely used. Screen recording 
analysis has been used to study user interaction with technological devices (e.g., Kim et al., 
2020), user experience in educational software (e.g., Lahav et al., 2020), and language learning 
or translation (e.g., Mueller-Spitzer et al., 2018; Orrego-Carmona et al., 2018). In a recent meta- 
analysis of multimodal research studies, only four of the 207 articles used screen recordings as a 
data source (Noroozi et al., 2020). Many of the studies that use screen recording techniques to 
capture data use quantitative approaches to data analysis and very few are done outside of 
controlled settings. Our method used an embodied discourse approach to data analysis in 
ecologically valid settings of students’ homes. Most research on adult learners relies on self- 
report which can be problematic as student perceptions and recollections may not accurately 
reflect what happens during the learning process. By combining subjective data from student 
reports and objective data through screen recordings, we were able to corroborate findings. 


Why Screen Recording Analysis is Important 


Using a multimodal approach to data collection afforded us the opportunity to watch 
participant facial expressions and gaze while simultaneously observing actions on their computer 
screen. While participants were recording their study sessions in the comfort of their home we 
could also hear the sources for distraction. We witnessed if a noise actually distracted their work 
or if they were able to work through the din. Observing mouse movements, eye movements, and 
hearing students speak to themselves out loud are artifacts we would not have been able to 
observe concurrently using other data collection methods. 
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Methods 


Participants 

Participants in our collective case study (Merriam & Merriam, 1998) were six 
academically high-achieving working mothers with prior experience in distance education. 
The students were studying at both the undergraduate and graduate levels. All participants were 
juggling multiple roles, employed outside of the home and had school-aged children. Initially, 
when we sought out participants, we were recruiting any adults currently in distance education. 
However, due to the small number of respondents, we were left with six participants who 
coincidentally were all high-achieving working mothers. 


Data Collection 


Our data collection methods included interviews with faculty of the targeted courses, 
interviews with student participants, recorded study sessions, debriefs with participants on these 
sessions, and student reactions to selected clips of study sessions. Interviews were conducted 
with each faculty member. These interviews were recorded, transcribed, and coded for common 
themes relating to instructor perceptions, interactions between students and the instructor, 
instructional design, and growing into the role of an online instructor. 

Interviews were also conducted with each participant at the start of the study. Students 
were read a statement about the goals of the study, were asked for recording permission, and 
whether they agreed to participate. Upon agreement, initial interviews were recorded. Questions 
relating to students' assets and challenges, time and place, and learning strategies were asked. All 
of the participant interviews were recorded, transcribed, and coded. Upon completion of the 
initial interviews, students were provided with instructions for video screencast recordings. 

Participants were asked to record at least 30 minutes of study sessions over the next three 
weeks at a rate of about one per week. Each participant had their own individual Zoom link that 
would automatically upload the recording to our private, secure Kaltura account. These accounts 
are through our University, are password protected, and are FERPA compliant. Students were 
prompted to share their screen and ensure their audio was on. Students were asked to also record 
these sessions even if they were not actively using their computer for the study session. For 
example, one student was reading from a text and taking notes but we were still able to capture 
her study session because we were recording with her camera. 

After each study session was recorded, a 15-minute debrief was conducted with each 
participant. We questioned students about what the recording was capturing, what went well for 
them during the week academically, and what posed a challenge for them. We also asked what 
tasks they completed for coursework during the week that were not recorded in the screencast. 

Throughout analysis of the screen recordings, we found that questions arose about what 
was happening in the recordings: What were students doing and why? How did that distraction 
impact their ability to complete a task? How did they work through that distraction? To gain 
more insight into the recordings, we compiled a few short video clips from study sessions for 
each participant. We then met with the participant to show them the short clips, get their 
reaction, and ask questions regarding the clips. 
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Data Analysis 


Macro-Level Review. We utilized a whole-to-part inductive approach (Erickson, 2006) 
to screen recording analysis, watching all of the recorded sessions to identify a construct for 
further analysis. Each researcher watched at least two recorded study sessions in whole from 
each participant, noting observations and questions for further analysis. For example, some 
questions that arose while we were watching the videos included: Does the amount of content on 
the screen impact how efficient the learner is with locating the information they need? Is on- 
screen behavior different when the student is familiar with the resource? How long does it take 
students to read text on the screen? What factors impact this reading speed? LOTS of 
distractions here. Are they different from the distractions of other students? After watching the 
videos and discussing our observations, we were all struck by the overwhelming amount of 
distraction that these students experienced during their study sessions in their homes. Distraction 
became the construct on which we focused on our analysis. 


Video Annotation. Our next step in data analysis was to begin annotating each study 
session video. We played, paused, and rewound video segments frequently to capture the actions 
we saw on the screen and in the environment, as well as the audio, noting start and stop times. A 
discourse analysis approach helped us to describe the actions we observed visually and 
auditorily, while embedded within the natural context that provides meaning to the actions 
(Hardy et al., 2004). A new excerpt was created in the annotation every time the action changed 
on the screen or there was a new interruption, which determined our units of study. Each excerpt 
was labeled as “un-distracted” or “distracted” to identify whether a distractor was present, 
whether or not the student appeared distracted by it. We used evidence in our observations to 
justify any inferences we made. For example, while we may not have seen a cell phone in the 
video window, hearing an alert tone followed by the student looking down for several moments 
led us to conclude that the student was looking at a message on their phone. Our initial 
annotations were created in tables in Microsoft Word. Each researcher watched some of the same 
study sessions and annotated them so that we could come to agreement on what we were seeing 
and calibrate our strategy for documentation. During the annotation process, we began to note 
that there were different types of distractions and different sources. 


Video Coding. After we had completed annotating several videos, we began to develop 
a coding scheme to analyze each study session moment-by-moment. Initially, we categorized the 
type of distraction the learner experienced, the type of activity interrupted, and source of the 
distraction. This was done by moving the tables from Microsoft Word to Microsoft Excel and 
adding columns for each distraction type (switching windows/scrolling, cell phone, social media, 
looking away from screen, engaging with other person, walks away, drinking/eating, movement, 
talking, noise, technology issue), the type of activity the student was engaged in (reading, 
writing/creating, watching video, navigating, self-regulation), and the source of the distraction 
(self, adult, child, animal, other). An x was placed in the corresponding column to mark each of 
these three categories for each excerpt (see Figure 1). In order to facilitate collaboration on these 
files, we moved the files to Google Sheets. One of the videos was coded by each researcher 
using this template to determine inter-rater reliability and calibrate our interpretations of the 
codes (Miles et al., 2020). As new codes were added, such as technology issue or self-regulation, 
we went back to re-code the completed videos to include these new codes. During this phase, we 
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began to realize that some distractors did not seem to affect the students and they remained 
engaged in their work. Our constant comparative approach and the iterative nature of our coding 
led us to notice what we initially overlooked (Glaser, 1965). We then added a new category of 
engagement (continues working, stops working, undetectable) and went back to recode each 
excerpt. View the data analyzed in this study here: https://tinyurl.com/3m2ycek8. 

Figure 1 

Screen Recording Analysis Coding Template 
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Quantitative Analysis. Adding codes and timestamps allowed us to conduct quantitative 


analysis on individual study sessions and then collectively to determine the frequency and source 
of distractions and to determine which activities were most engaging. We developed a list of 
questions to ask using the data we collected. Those questions and the results are listed here. 


1. 


2. 


What was the ratio of time that students endured distraction during study sessions versus 
being distraction free? On average, distractions were observed during 34% of the recorded 
study sessions. 

How did students spend their time on the different types of learning activities? Students spent 
the majority of the recorded study time writing/creating (53%) or reading (36%), with the 
remaining time spent watching videos (22%), navigating (22%), or self-regulating their 
learning (6%). Sometimes students work engaged in more than one activity at the same time, 
so these values do not total to 100%. 

How often did students stop working and for how long? Students stopped working for 21% of 
the recorded study time, remained engaged in work for 72% of the time, with the remaining 
6% undetectable. Students stopped working an average of every three minutes and thirty- 
eight seconds. 

Which distraction types were associated with students stopping (or continuing) working? 
Levels of engagement during distraction varied by distraction type (see Figure 2). Students 
continued to work during 60% of the time where there were extraneous noises, 41% of the 
time when there was talking, 40% of the time during technology issues, 34% of the time 
while eating/drinking, 23% of the time when there was movement observed, 19% of the time 
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while they were simultaneously engaged with others, 17% of the time while they were 
looking away from the screen, 16% of the time while they were switching windows, and 4% 
of the time while they were using a cell phone. They disengaged from their work completely 
when they were using social media or walked away from their computer. 


Figure 2 


Percent of Time Engaged During Distraction by Type 
0% 10% 20% 30% 40% 50% 60% 70% 


Noises 

Talking 

Technology issues 
Eating/drinking 

Movement 

Engaged with others 
Looking away 

Switching windows/scrolling 
Cell phone 


5. Which learning activities were associated with students continuing (or stopping) working? 
Students appeared to remain engaged during distraction at different rates depending on the 
type of activity they were engaged in (see Figure 3). During self-regulated learning activities, 
students persisted through distraction and remained engaged 92% of the time, 87% of the 
time during navigation, 81% of the time while reading, 75% of the time while 
writing/creating, and 66% of the time while watching videos. 
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Figure 3 
Percent of Time Engaged through Distraction by Learning Activity 
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6. Who/what was the greatest source of distraction? Students were their own greatest source of 
distraction, either by engaging in a different task other than the learning activities or 
choosing to respond to a distractor such as a child or message alert. Students were the source 
of distraction for 24% of the total recorded study time, with distractions from children 
accounting for 17% of the recorded study time, 9% by other sources, 8% by other adults in 
the environment, and 1% from pets. 

7. Was the source of the distraction related to whether they continued working? The source of 
the distraction appeared to be related to whether the student continued to remain engaged 
(see Figure 4). Students were able to remain engaged during 75% of distractions from pets, 
55% of distractions from other adults, 47% of distractions from children, 45% of distractions 
from other sources, and only 16% of distractions where they were the source of the 
distraction. 
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Figure 4 
Percent of Time Engaged During Distraction by Source 


80% 
70% 
60% 
50% 
40% 
30% 
20% 
10% 


0% 
Pets Adults Children Other Self 


Triangulation. Results were corroborated with interview data and voiceover reflection 
by participants of selected clips (Beach et al., 2021; Sezen-Barrie et al., 2014). Interview and 
weekly debrief transcripts were analyzed to identify qualitative data that supported the 
observations we made in the recorded video sessions. For example, one student appeared to 
remain engaged during loud noises her spouse was making while he was playing video games 
right behind her during one recorded study session. In her weekly debrief, she explained that she 
also plays video games and was easily able to tune out those noises. Other students explained 
that their level of engagement during distraction also depended on their interest in the content 
being studied and how applicable and engaging the assignment was. 

After each student had completed recording at least three study sessions, we reviewed 
these recordings to select clips that we wanted the students to help us understand. We then met 
with each student again to play back those video clips. The reactions from the participants during 
this phase of the project included shock and disappointment. For example, in one student video 
reflection session, a student became upset while watching one of the clips we selected. Her son 
was very excited and interrupted her studies to tell her something. The student looked up briefly, 
saying, “Hey, you guys are triplets! Super cool. I love it,” then went right back to her work. 
While watching this clip, the student teared up and said, “That was my, ‘I’m really, really trying 
to engage with you and to give you what you need, but I can’t right now.” That was awful.” The 
participant was conflicted with doing her job as a student and doing her job as a mother and it 
was apparent in her emotional reaction to the clip. 


Discussion 


Results of our data analysis included high levels of distraction in the home during study 
sessions from children, spouses, and pets, with distraction experienced during an average of one- 
third of the total recorded study time. Interest in the content and instructional design were two 
factors that were related to maintaining engagement during distraction. Students remained more 
engaged through distraction during self-regulation, reading, and writing/creating activities and 
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responded more positively to courses that were systematically designed using effective practices 
in distance education. 

The quantitative analysis described here averages all of the study sessions from the six 
participants, which is a small sample size. There was much variability between the participants 
on the amount of distraction that they faced from a low of 10% to a high of 60% of a total 
recorded study session. Therefore, these results cannot be generalized to the whole population. 
However, the data described here does provide some insight into the experience of adult learners 
as they participate in distance education. 


Potential Concerns Regarding This Method 


Potential concerns rest in technological complications such as a participant’s camera not 
being turned on, their sound not working, or the participant not recording the study session. 
Because of the ecological validity that this method affords, sensitive events or information from 
the home environment of the participant may be captured in recordings that could cause distress 
for the participant. The screen recording only captures what we can see on camera. When a 
student reaches for something, has a conversation with someone off camera, or leaves the view, 
we do not know what is happening. Assumptions can be made, but at the risk of being 
inaccurate. 

Screen recording analysis is a very labor intensive methodology. Analysis of videos and 
multiple coding strategies took a lengthy amount of time. Tools that would assist in automating 
this process or speeding the process up would be beneficial. 


Suggestions for Using this Method and for Further Research 


The quality of the recorded video and audio will determine the ability to accurately 
capture events in the screen recordings, so effort should be made to work with participants to 
optimize equipment. Some troubleshooting may need to be completed with participants before 
they record their sessions. Management of media and coding files should be considered during 
study planning. Screen and webcam recording could be combined with other data from tools 
using Harvard’s Multimodal Toolkit (https://mmla.gse.harvard.edu), such as eye tracking or 
emotion detection; with physiological inputs like heart rate or electrodermal activity; and LMS 
data from log files to provide a more complete analysis of the learning experience. While our 
study helped to illuminate the experience of adult learners in distance education, we recommend 
using this method to study other populations, such as K-12 students in remote learning, 
traditional-aged college students, or other special populations. 


Conclusion 


The use of screen recording analysis in authentic environments as a methodology to 
understand the experience of adult learners in distant education is an innovative approach that 
proved beneficial in our research, however, we recommend future research in other settings to 
determine further benefits, feasibility, and limitations. The use of screencast, webcam video and 
audio recordings provided a modality for the researchers to collect data and analyze the learning 
techniques and distractions in the learners environment and how it influenced student learning. 
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This method in natural settings can be combined with traditional techniques, like participant 
report, to more fully understand the underlying experiences of the learner through screen 
observations. 
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Abstract 


Learners in higher education settings across the country bring to the classroom a wide array of 
educational experiences, cultural practices, and individual identities. The cultural differences 
learners carry into their online learning environments have the potential to influence how they 
interact with their classmates and instructors as well as how they interpret course content, 
learning objectives, and assignments (Chita-Tegmark, 2012). An asset-based approach to 
instruction frames the diverse nature of learner experiences as strengths and provides 
opportunities for learners to express their individual identities through choice (Lopez, 2017). 


Introduction 


Identity development can be viewed as the result of the interaction among 
psychological/individual factors, cultural factors, and contextual factors of a learning 
environment (Collins, 2018). Implementing an asset-based pedagogical approach can encourage 
learners to see themselves reflected in their work. Further, opportunities for learners to build 
upon prior knowledge can serve to validate their experiences and help develop and strengthen 
their identities as learners (Darder, 2012). Asset-based approaches to instruction view students’ 
prior knowledge and lived experiences as valuable tools that can be utilized to solve problems 
that are both meaningful and relevant to their lives. This instructional approach positions 
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students’ lived experiences, their interests, geographic locations, familial and cultural values, and 
social concerns as central components of the learning process (Wright et al., 2016; Yosso, 2005). 

Exposure to asset-based instructional strategies that are problem-based can encourage and 
support the identity development of all learners, particularly for those interested in pursuing 
careers where they are often underrepresented such as those in STEM fields. Problem-based 
learning challenges can provide educators an opportunity to not only leverage students’ social 
and cultural capital, but also develop agency and strengthen identity through the process of 
working on a project that is both relevant and meaningful to them (Freire, 1970). 


Purpose and Research Questions 


This study examined the ways in which learners express their identity and culture when 
provided opportunities for voice and choice in an online learning environment. The study 
addressed the following research questions: 


e When given choice in learning outcomes, in what ways if any do learners express aspects 
of self-identity? 

e How are opportunities for identity expression perceived by learners in an online learning 
environment as they relate to agency, voice, and choice? 


Methodology 
Participants 


Participants (V=105) were a convenience sampling of learners from three sections of an 
introductory education technology course offered in a college of education at a large public, 
southeastern university. The first two sections were held in Fall 2020, and the third section was 
held in Spring 2021. All participants were enrolled in undergraduate education degree programs, 
including early childhood education, elementary education, and exceptional education. The 
course titled Introduction to Technology for Educators included modules related to classroom 
management tools, multimedia, communication networks, interactivity, educational software and 
legal, ethical and social issues related to educational technology tools. The purpose of the course 
was to prepare learners to be competent computer-based technologists. Throughout the course, 
students learned how to successfully integrate instructional technology tools into their teaching 
approach and cultivated their own sense of the importance of effective instructional technology 
modeling as part of their teaching and learning strategies. 


Procedure 


This study examined data collected from an introductory education technology course for 
undergraduate learners. In the first module of the course, learners were asked to complete a 
digital T-shirt assignment to introduce themselves to their classmates. The purpose of the 
introductory digital T-shirt assignment provided learners an opportunity to utilize technology to 
create and present T-shirts that incorporated images they felt represented salient aspects of their 
identity. The importance of each image was described by the learners and a rationale for the 
images, words, and hashtags included on the digital T-shirt was posted in a public discussion 
forum. Learners in the class were asked to comment on the work of other classmates. The 
submissions from this assignment were used as the baseline to determine how learners self- 
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identified aspects of their identity such as their personality traits, ethnicities, religions, hobbies, 
and family structures (see Figure 1). In addition to the creation of a t-shirt, learners were asked to 
post a response that included what their image represented about them and if they had a hashtag 
to describe themselves, what would it be (see Figure 2). Learners were also asked to respond to 
at least two other classmates. 


Figure 1 


Example of student T-shirt Image 


The next opportunity for the expression of student identity occurred during the second 
module of the course. Learners were tasked with creating their own digital Escape Rooms. This 
activity required learners to develop and create their own digital escape room based on a topic of 
their choosing (see Figure 3). Learners then shared their work with others in the course as well as 
friends, coworkers, and spouses to receive feedback and revise if necessary. At the conclusion of 
the assignment, learners reflected on their experience creating an escape room and how they 
anticipated this activity could be used in the future. They were asked to respond to the following 
questions: 


What did you learn as a result of completing the digital escape room? 

How can you use this innovative assignment now as a student? 

How will you use this assignment in the future with your own classroom? 

What did you improve as a result of the test/retest step in the creation of the digital 
escape room? 
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Figure 3 


Examples of students’ digital Escape Room Submissions 


Slide 1 Slide 2 


Slide 3 Slide 4 


CURTIS LEMP 


SOME LIKE IT HOT 
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Slide 1 Slide 2 


SLY 


You are leaving your grandparents house in 30 
minutes! Quick, press play and break the code in 
time before the clock runs out. You must see what's 


insidel! 


Civil War Escape Room 


Student Name 


Slide 3 Slide 4 


You are at your grandparents 
house and exploring through 
their bookshelf. You come 
across their parents, your 
great-grandparents, photo 
album. The outside looks old 
and worn and you move to 
turn the first page, however 
there is a lock on the photo 
album. You must explore 
and crack the codes in order 
to gain access to the photo 
album. 


You begin to shake the album trying to open it, out of the 
small crack falls a newspaper clipping. Read the clipping 


for clues! 

— WAR DECLARED! 
org/earlyamericannewsmed) yo oa a 
Jexhibits/show/news-and: UNE WF TOE STE ST, Mua 
he-civil-war 


Link to the locks on the next page! | 


Data Analysis 


After conducting a comprehensive literature review related to topics such as culturally 
relevant instructional approaches (CRIA), asset-based pedagogy (ABP). voice and choice in 
online learning environments, research questions were developed. In order to answer the 
identified research questions, a descriptive content analysis was conducted. First, learner 
assignments were curated from the Learning Management System (LMS). Data that met the 
following criteria were considered for analysis (a) submissions that evidenced learners' self- 
described identity in initial activity, (b) submissions that incorporated learners' identity and 
culture in the second activity and (c) reflections that included learners’ perceptions of the 
affordances of voice and choice in an online learning activity. 

Next, submissions, artifacts, images, and reflections were coded, and a coding frame was 
constructed based on the data. After initial data analysis and coding commenced, general 
variables were determined by the research team to be included in the coding scheme. These 
included aspects of identity such as: (a) family, (b) country of origin, (c) faith, (d) personal 
affiliations, (e) affections, and (f) popular culture. Evidence of agency such as self-efficacy, 
choice, and collaboration were also considered as these themes frequently found throughout the 
data. Hashtags included as part of the t-shirt assignment were also collected and analyzed (See 
Figure 3). The following themes emerged through the coding process: (a) perseverance, (b) 
personal identity, (c) positivity, and (d) authenticity. 
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Figure 3 


Student-generated Hashtags 


#yougotthis 
#nevergiveup 
#ambitious 
#getitgirl 
#queenoffreshstarts 
#determined (5x) 
#motivated 
#perseverance (2x) 


#strong 


#gettoknowhannah 
#hippieteacher 
#Articulate360Newbie 
#organized 
#Nurturing 

#lucky 

#wildchild 
#SweetAsPie 
#readmyface 


#SapereAude (dare to 
know - learning and 
striving in education) 


Positivity 
#go00dvibes 
#CAREFREE! 
#Everythingisokay 
#bright 
#cares 
#passionate 
#blessed (4x) 
#easygoing 
#empathetic 
#ChooseJoy 
#erateful (2x) 


Authenticity 


#beyourself 
#Natural 
#beintentional 
#truthful 
#compassion (2x) 
#authentic 
#authenticity 
#BeHonestAndReal 


Findings and Discussion 


Based on the coding frame, 29% of the potential responses were related to self-identity 
and 27% were evidence of learners’ acknowledging that they had a voice and choice in choosing 
the direction of their learning. Data from this study demonstrated that learners frequently made 
efforts to connect their work with their own experiences and funds of knowledge. When learners 
were given agency and the power to determine the topics for their assignments, many noted in 
reflections that it improved their learning experience. One learner made the following comment: 


The assignment was nice in that we were allowed to choose any moment in the United 
States of America to choose from for our escape rooms. I was glad we could be creative 
in choosing topics, riddles, and questions for our escape rooms. I look forward to making 
many more escape rooms! 


The introductory T-shirt assignment detailed in this study presented a learning context 
that utilized technology to provide opportunities for learners to authenticity express their 
identities to their peers. Learners oftentimes chose to weave aspects of their identities into 
assignments that allowed for voice and choice throughout the course (See Figure 3). 
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Figure 3 


Example of student identity across assignments. 


#MOTIVATE 


AMERICAN 
STORY 


PLEASE PAY ATTENTION TO THE HINTS INCLUDED IN SOME OF THE QUESTIONS! Information for the questions is linked down below 
each question. 


Escape Room 


* Required 


You are now in a digital escape room. You are being tested on some important 
events in LGBTQ+ American History in order to win a trophy at the end of the last 
room. In each room you enter, there will be a clue to help you unlock the door to 
the next room. After you finish the three questions you will be let out of the 
digital escape room and you can collect your prize! Enter your name below to 
start." 


Your answer 


Next 


Never submit passwords through G: 


GoogleForms This content is neither created nor endorsed by Googie 
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In addition to aspects of identity and agency evidenced in assignment submissions, 
learners also frequently discussed how the opportunities for voice and choice embedded in the 
tasks positively contributed to their self-efficacy, creativity, and overall engagement in the 
course. The student who created the examples in Figure 3 reflected on the assignment and 
commented: 


The things I put on my shirt are what make me, me! On the left sleeve I have a rainbow 
heart because I am a very proud member of the LGBTQ+ community. This is something 
that is very important to me not only because it's a community that I’m apart of but also 
because I always want others to know that I will be supportive of them and that I am 
always a trustworthy person that they can come to. On the right sleeve of my shirt is a 
picture of my family and I because they are my everything. The top of my shirt has a 
headphone with music coming from it because music is my escape and something that 
always helps me when I’m feeling down, or even just happy! Below that is the beach 
because when I’m not doing school or watching my sister play softball, I’m definitely at 
the beach! .... And lastly, the hashtag I would use is #beyourself. I have struggled a lot 
with knowing when to be myself and when to kind of hide who I want to be from others 
but I'm proud to be me and I always want to be myself! 


The Escape Room Breakout challenge provided multiple examples of identity expression 
as students choose which topics they integrated into their digital projects and the images and 
storylines they included. The Escape Room challenge utilized a problem-based instructional 
approach to provide students with an opportunity to apply their personal knowledge and lived 
experiences to design a problem, develop a storyline, work towards finding solutions, and reflect 
on the process as a whole. One learner remarked: 


While forming questions I would switch back and forth between the slides and the form 
to make sure everything lined up well and decided to make my last question about 
LGBTQ+ rights because those are especially important to me as a queer woman. So, I 
found a website that showed me a map of the penalties/protections different countries 
have in place for members of the LGBTQ+ community and asked participants to pick 
which two continents had the harshest consequences according to the map. 


Providing opportunities for identity building experiences in higher education courses can 
encourage meaningful participation in course activities. The inclusion of problem-based learning 
challenges that utilize an asset-based approach can cultivate deeper learning by centering 
problems within a context that is meaningful to students (Baeten et al., 2010). Several 
participants noted their intention to incorporate asset-based instructional approaches with their 
own students in the future. One student noted: 


This digital escape room had myself, as the creator, thinking outside the box and using 
my creativity to enhance the design of it. This project expanded my knowledge in various 
ways, and I am becoming more comfortable in using these particular technological 
platforms. 
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The use of technology tools can support authentic identity expression and encourage 
learners to make connections to their own experiences and the experiences of others in the 
context of the course. Identity expression was evidenced across multiple assignment submissions 
throughout the course and students noted how the opportunities for voice and choice positively 
impacted their motivation, self-efficacy to use technology tools, as well as overall impression of 
the course. 
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Introduction 

Entrepreneurship is defined as the ability to turn an idea into an action using creativity to 
create value for others (Chen et al., 2001; McGee et al., 2009; Tsai et al., 2016). It is the driving 
force of our economy. Arming young people with an entrepreneurial mindset will promote future 
business creation and build a pipeline of entrepreneurial thinkers. Without the innovation and 
risk-taking of entrepreneurially minded individuals, we would not invent, empower, or thrive as 
a nation. Also, an entrepreneurial mindset helps increase creativity and self-confidence in every 
aspect of people's lives and is an essential driver of growth and sustainability (Neck & Corbett, 
2018). 


Entrepreneurship Education 

Entrepreneurship education is needed to provide the necessary knowledge and skills for 
young people to launch innovative practices to be competitive in the world (Gargouri & Naatus, 
2019). Entrepreneurship education provides students with a safe environment to experience what 
it is like to be entrepreneurs (Neck & Greene, 2011). Unfortunately, entrepreneurship education 
is underdeveloped in K-12 schools (Zhao, 2012a), even though teaching entrepreneurship in the 
K-12 sector can help build a generation of young people armed with an entrepreneurial mindset 
and become innovators, creators, and leaders in the workforce. 


Purpose of this Paper 

Teaching entrepreneurship in the K-12 sector can help build a generation of young people 
armed with an entrepreneurial mindset who become innovators, creators, and leaders in the 
workforce. The purpose of this presentation is to provide guidelines to integrate entrepreneurship 
into K-12 classrooms. 


Guideline 1: Use a design-based thinking approach 

Design-based thinking (DBT) is a cyclical process of observation, synthesis, generating 
alternatives, critical thinking, feedback, and creativity, similar to the entrepreneurial process 
(Daniel, 2016). The entrepreneurial process emulates the design-based thinking process, as they 
are both iterative and start with problem identification moving from ideation/brainstorming to 
prototyping and testing ideas (Neck & Greene, 2011). By approaching entrepreneurship from a 
DBT perspective, teachers can ask students to identify problems they want to solve, generate 
ideas, gather information from various sources to solve the problem, and launch their business. 
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Guideline 2: Promote students' entrepreneurial mindset 

An entrepreneurial mindset is the state of mind that changes an individual's status to an 
entrepreneur. An entrepreneurial mindset concerns the analysis of the world, its opportunities 
and possibilities, and the understanding of how an individual can contribute to the progress of 
economic and social systems (Kouakou et al., 2019). An individual with an entrepreneurial 
mindset has a set of skills enabling him/her to recognize opportunities/identify problems that 
need to be solved, take risks, create value from opportunities, and overcome challenges (Fayolle 
& Gailly, 2015). Empirical studies show that an entrepreneurial mindset is a precursor for 
entrepreneurial behavior, intentions, and actions and indicates an individual's worldview (Daniel, 
2016; E. Kim & Strimel, 2020; Korte, 2018; Kouakou et al., 2019; Popescu, 2014; Rae & 


Guideline 3: Experience the entrepreneurial process from inception through 
implementation 

This guideline is grounded on the experiential learning model. Kolb's experiential 
learning model focuses on experience as the main force driving learning because "Learning is the 
process whereby knowledge is created through the transformation of experience" (1984, p. 38). 
According to Kolb (1984), a person learns when he or she can progress through a cycle of four 
stages, including a concrete experience, a reflection on the experience, an analysis of the 
reflection, and an application of that analysis to future experiences. Empirical studies show that 
an experiential learning approach contributes to students’ entrepreneurial mindset. For example, 
Bell (2015) found in her research that an experiential learning approach requiring students to 
pitch ideas, present findings and reflections, and work with community members while 
developing a business idea increased entrepreneurial mindsets. Lackéus (2020) found that 
experiencing a real-life business startup process increased student self-efficacy for 
entrepreneurship. 


Guideline 4: Provide scaffolding 

Vygotsky's Zone of Proximal Development (ZDP) provides underpinnings for this 
guideline. The ZPD is a construct that bridges the gap between what a student can already do and 
what he or she can do with assistance. The ZPD is the target area where scaffolds are the most 
helpful. Instruction focused within a students' ZPD provides just enough challenge to help them 
enough help to be successful while still challenging them. As students gain skill, teachers 
decrease scaffolding support until students can accomplish the task independently (Belland et al., 
2013; Vasconcelos & Kim, 2019). 
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Entrepreneurship is inherently ill-defined as the entrepreneurial journey has many twists 
and turns and uncertain outcomes, providing challenges. For students will little or no experience 
in entrepreneurship, this process may seem intimidating. With appropriate scaffolding, students 
can progress so that scaffolds can be faded as students gain competence and confidence (Belland 
et al., 2008). Scaffolding strategies include highlighting essential task elements, questioning, 
modeling how experts solve problems, providing feedback, etc. (Belland, 2014). Teachers can 
scaffold entrepreneurial learning by modeling their thinking when describing new concepts. 
They can lead by example, demonstrating their thoughts and actions in the face of new 
information. 


Guideline 5: Use authentic problems 

Teachers’ objective is to develop ways for students to experience the realities of starting a 
business in a risk-free or safe environment such as the classroom (San Tan & Ng, 2006). This 
way, entrepreneurial competencies can be developed that actively engage learners (Macht & 
Ball, 2016; Miles et al., 2017; Robinson et al., 2016). Authentically aligned activities should be 
used to enable the practical application of the same skills knowledgeable professionals apply 
when addressing similar activities (Fook & Sidhu, 2010; Macht & Ball, 2016). Authentic 
activities enable deeper learning by increasing students’ motivation and engagement and 
supporting students' achievement. For entrepreneurial education, teachers can ask students to 
identify local problems, interview stakeholders to improve their business ideas, create a product 
to conduct experiments to test the product, and pitch their ideas to an authentic audience. 
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Abstract 


Computational thinking (CT) has become an integral 21st Century skill that facilitates 
problem solving across disciplines (Grover & Pea, 2018). Thus, embedding opportunities and 
scaffolding for CT learning within K-12 education is now a focus for scholars and practitioners 
(Hsu et al., 2018). Recent progress in this area has primarily focused on middle and high school 
levels, where many students now have increasing opportunities to learn CT through designated 
computer science and STEM courses, while K-5 educators often lack the tools to support student 
learning of CT skills and practices (Code.org Advocacy Group, 2018). Additionally, an 
interdisciplinary approach to CT teaching and learning can be more effective than having 
students learn these skills in isolation from other subject matter (Sandford & Naidu, 2016). 

K-5 teachers may be at an advantage for integrating CT across disciplines (e.g., ELA, 
math, science) and creating robust CT learning experiences, since they teach a variety of subjects 
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to the same students each day. However, there is a dearth of research that has focused on specific 
learning technologies and their respective in-platform supports that would help teachers 
seamlessly integrate CT into disciplinary content learning (Kale et al., 2018; Pila et al., 2019). In 
response to this need, we are iteratively designing and developing a narrative-centered digital 
learning environment to engage upper elementary students in computationally-rich science 
learning. Digital narrative creation offers students an interactive learning experience and enables 
the creative exploration of scientific phenomena (Henriksen et al., 2016), while also reflecting 
many CT concepts in the writing and story creation process (Parsazadeh et al., 2020). 


Methods 


To gather feedback from upper elementary teachers about our narrative-centered digital 
learning environment and its usability for classroom and distance learning settings, we conducted 
an online focus group in the fall of 2020 which examined teachers’ perspectives on the learning 
environment’s facilitation of story creation using custom narrative blocks as well as the efficacy 
of its integration of science, English language arts (ELA), and CT concepts. Our protocol 
included three components: 1) an introduction to the study and learning environment; 2) teacher 
experimentation with the learning environment; and 3) a post-experimentation focus group 
interview. The learning environment used in this study, INFUSECS, is designed to enable upper 
elementary students to create interactive digital stories and utilizes a custom-built narrative 
programming environment, where students use a block-based programming interface to create, 
revise, and visualize interactive narratives. 

After a 15-minute introduction and icebreaker activity to help establish rapport within the 
group, participants were introduced to the overarching goals of the project and learning 
environment. The teachers were then randomly assigned to Zoom breakout rooms where they 
had 20 minutes to experiment with the learning environment and then share their perspectives on 
the learning environment, a narrative planning worksheet designed to facilitate story creation and 
embedded activities. A researcher observed each breakout room, answered participants’ 
questions, and addressed any technical issues that arose. At the end of the think-aloud breakout 
sessions, researchers sought participants’ permission to collect screenshots of the computational 
artifacts (Figures 1 and 2) produced during the sessions. Participants and researchers then 
rejoined the main Zoom meeting where participants shared their perspectives regarding the 
learning environment activities through open-ended interview questions. This portion of the 
focus group session lasted approximately 25 minutes and the interview questions examined 
participants’ perspectives regarding the learning environment’s ability to facilitate story creation, 
its usability, and the efficacy of its content integration. 

The researchers held a debriefing session after the focus group to discuss and record field 
notes. The Zoom sessions, including breakout rooms, were transcribed and divided among the 
researchers for qualitative thematic analysis. The researchers discussed the qualitative results to 
reach consensus on the thematic elements of the data. Themes that emerged from the analysis 
provided impetus for refinements to our digital narrative-centered learning environment. These 
results are presented in the following sections corresponding to the overarching themes. 
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Figure 1: Narrative Designer Program Editor 
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Participants 


During fall 2020, researchers conducted an online focus group with four 4th grade 
teachers. The sample consisted of instructors from Northern California, including 3 female and 1 
male teacher. The participants were a convenience sample made up of 4 local retired teachers. 
All instructors had prior ELA and science teaching experience, with 75% of the teachers 
planning to cover physical science in their classrooms during the school year. 

Researchers provided participants with a prior experience survey consisting of 3-point 
Likert items, ranging from “None” to “A Lot.” Seventy-five percent of the teachers reported 
having “Some” level of experience using coding activities in their classroom. Twenty-five 
percent reported having “A Lot” of experience using digital narratives, while the remainder 
reported having no experience with digital narratives. Researchers collected data via the online 
Zoom video conference platform using the tool’s main and breakout room functionality. 


Findings 


Our analysis suggested that teachers believed that the learning environment would 
facilitate creative digital storytelling with its custom narrative blocks and provide an engaging 
environment for students to learn science, ELA, and CT. However, teachers experienced some 
confusion in getting started in the integrated coding environment and difficulties with its 
accompanying instructional materials, which suggested that additional navigational and 
instructional supports were needed in the learning environment. Further, the focus group results 
prompted us to give more consideration to providing a better balance for the integration of the 
conceptual knowledge we aim to support in physical science with this platform. The following 
sections provide more detail on the overarching themes that resulted from our analysis. 


Facilitating Story Creation with Narrative Blocks 


A key aspect of the learning environment is to facilitate students’ abilities to develop 
science-based digital narratives using custom narrative code blocks. Thematic analysis revealed 
that the learning environment’s custom narrative blocks aided teachers’ development of their 
digital stories. Teachers referenced the custom narrative blocks as they encountered them in their 
exploration of the learning environment and this seemed to spur the creation of their narratives. 
Teachers were also easily able to discern that story characters could be added using the 
character-focused custom blocks and without further prompting from the researchers, eagerly 
began incorporating dialogue blocks for their characters’ interactions. 

Teachers seemed to perceive the custom narrative blocks as intuitive and we observed 
that both groups of teachers spent the majority of their breakout session on the character dialogue 
of their digital stories. Despite some expressed frustration with typing their characters’ dialog, 
teachers remained engaged in creating their science-based narratives. However, one point of 
contention seemed to arise from the appearance of characters not matching teachers’ 
expectations of how their characters should look. A teacher in one group fixated on a male 
character offered in the learning environment that was dressed as a nurse despite incorporating 
the character into their story in another role. The teacher expressed some disappointment with 
the inability to change the character’s appearance to match their expectation for the role they 
were assigning, but in this early stage of the learning environment development, the characters 
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provided were not customizable. Finally, during the post-interview, teachers expressed that 
timing would be a key element to using the learning environment and associated activities in the 
classroom. The learning activities were perceived as enjoyable and useful to their students, but 
teachers suggested that they would need to spend some instructional time on ELA concepts 
before jumping into the learning environment for digital story creation. One teacher expressed 
the need to “lay out [a] lot of groundwork” and another expressed that: 


“T think you'd have them work in teams and plan out the story? And they'd have 
to get some ideas, I think. So, you'd have to brainstorm, like she said and I'd 
see this being at least [a] week too and using a lot of language arts time before 
they dive into it. What a story should look like even.” 


Usability of the Learning Environment 


Data from the usability study indicated that teachers felt the INFUSECS learning 
environment operated according to specification. Specifically, teachers like the platform and 
reported that it was engaging and easy to use. While teachers needed some assistance with 
operating the technology or completing the narrative planning worksheet, teachers found that, 
overall, there were no issues with the fundamental platform operations, including coding block 
accessibility or the dragging of blocks and attachment of blocks. 

Teachers were able to intuitively and cooperatively use the narrative blocks and planning 
document framework. The participants worked together to fill out the planning document and use 
the learning environment. The planning document ‘dialog’ and ‘ask the audience a question’ 
organization scaffolded the teachers’ thinking during the story design process from a beginning 
stage, through to the middle and end. The platform code categories and naming of the custom 
narrative blocks helped teachers to identify the story creation components needed to develop 
their planned story: 


“All right. Stage direction left, right, middle, Hailey exits, Hailey enters. Oh, 
Hailey enters stage left. Oh, I got it. I got it. I got it. Hailey enters stage left. 
Dialog, Hailey’s going to say ‘Oh my geezy, is everyone okay?’” 


Dragging the blocks onto the main work area was also seamless and the teachers easily attached 
the blocks together to create their narrative programs. Half of the teachers reached the point of 
running their narrative programs and seeing the corresponding narrative visualization, and when 
directed to, successfully observed the translation of their story into visual form. 

Teachers did, however, have some difficulties. Participants did not grasp the interactive 
theatre nature of the platform and thus were not sure how to properly structure a ‘theatre 
audience’ question and response block. Participants also could not successfully edit the blocks in 
the question space of the planning document. The participants tried multiple times to click the 
question box and enter text, but the image formatting did not allow text entry. Within the 
learning environment, two teachers had initial confusion about what to do when they first opened 
the software. As a result, they sought researcher direction and support. In another portion of the 
session, half of the teachers had some difficulty navigating the integrated coding environment 
because they did not understand how to close/exit the code category sections of the workspace. 
One teacher expected the coding environment to be platform agnostic and unsuccessfully tried to 
access it using an iPad. Despite these challenges, with a small amount of feedback from the 
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researcher and additional time, the teachers were able to move through each initial source of 
difficulty without further assistance. 


Content Integration Efficacy 


Our focus group findings indicated that the participants were able to make more explicit 
disciplinary connections to ELA in comparison to science and CT. Several teachers indicated 
that they would specifically dedicate ELA instructional time for the activities, in particular, 
allowing students a few days to draft and build their stories. One teacher felt the learning 
environment would be well-suited as a tool for generating the creative writing genre of 
playwriting and dialogue between characters while others remarked at how the built-in scenery 
of the narrative environment such as the waterfall could be sources of energy on the island. 
However, in the post interview they expressed some concern that students would need group 
brainstorming sessions to make these explicit science connections. 

Finally, our investigation probed the participants to discern if they noticed potential 
opportunities for CT integration. Although teachers in our study never explicitly named CT 
concepts as a part of the learning experience, their practices exhibited CT elements that could 
potentially be integrated into their pedagogy with the platform through professional development 
and training. Moreover, the teachers suggested that graphic organizers could help students 
decompose and abstract the necessary story elements and scientific components that would be 
needed to compose their narratives. During the interview, teachers shared ideas to consider for 
the learning environment that aligned with CT. For example, one teacher noted, “I think you'd 
have them work in teams and plan out the story,” indicating he saw the environment fostered 
opportunities for collaboration. Another teacher discussed the learning environment’s value for 
creating artifacts: 


“At the end when we saw what we created, that's why I feel like my kids would 
really like it. Because I see what the end could look like and if they can see what 
the end would look like, I think they would go crazy wild because it's great.” 


Our observations of teachers’ interactions with the environment also indicated that our 
participants saw the animation tool and the programming blocks as means for fostering CT 
concepts and practices like debugging, tinkering, and evaluation as we witnessed them engaged 
in these processes. 


Iterative Development and Future Work 


To address concerns raised during the focus group, we implemented an initial set of 
iterative refinements to the INFUSECS learning environment. The first refinement aimed to 
improve usability of the software. Upon logging in, users now encounter an overview map 
(Figure 3) highlighting key navigational features and an introductory video sequence (Figure 4) 
to help each participant connect the individual pieces of the environment with the overall goals 
and directives of the platform. 
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Figure 3: INFUSECS Overview Map 
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The second refinement was the incorporation of the Science Content Explorer. This 
component teaches foundational energy concepts (Figure 5), engages users with an interactive 
simulation of energy conversion methodologies (Figure 6) and provides proximal learning 
opportunities through sense-making questions. 

The interactive simulation feature also aims to bridge the gap between the learning 
environment’s science learning objectives and the expression of energy conversion principles 
within students’ digital narratives. Finally, a set of story starter blocks were added as a scaffold, 
enabling users to play a simple story animation and familiarize themselves with the instantiation 
and design of digital narratives. 

Building on these refinements to the learning environment, future work will include 
conducting classroom feasibility studies. This will include both remote and in classroom studies 
as in-person learning becomes safe for students, teachers, and researchers. Feedback gathered 
from these studies will drive further refinement of the learning environment. 


Figure 5: Science Content Explorer Foundational Energy Concepts 
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Figure 6: Science Content Explorer Interactive Simulator 
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Abstract 


In this paper, we describe the design and development of a formative assessment plan with a 
mastery-based grading approach, implemented in one college-level math course offered during 
the pandemic, to reimagine the purpose of assessment in transforming exams into learning 
experiences. A retrospective study of this implementation indicates that this assessment strategy 
has psychological and academic impacts on student achievement. It enhances student learning by 
changing the negative perceptions of failure on assessments into an opportunity for growth. The 
outcome of the study also demonstrates how different groups across the university work 
collaboratively to innovatively implement teaching and learning strategies with effective 
pedagogical practice to promote student success. The working model can be generally applied to 
varying disciplines to support educational transformation in the post-pandemic era. 


Keywords: mastery learning, mastery-based grading, assessment, mathematics education 
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Background 


The Impact of the coronavirus pandemic on higher education 

Institutions of higher education and enrolled students experienced substantial difficulties last 
year due to the COVID-19 pandemic. In response to this crisis, the shift to online learning 
preserved academic continuity. While this transformation has been challenging, much has been 
learned from this experience that can guide educators to rethink teaching and learning through 
the exploration of new pedagogical strategies and philosophies to progress through 
unprecedented educational disruption. 


The quick turn-round to an online mode of delivery raised many concerns about the quality of 
education being offered. For example, the use of conventional assessments with high-stakes 
examinations to test remote learners lacks the flexibility that frustrated both instructors and 
students during the pandemic (Fuller, et al., 2020). High stakes testing has also been questioned 
in regards to the limitations of reliability and stability in accurately assessing student learning 
(Knight, 2002; Zimmerman & Dibenedetto, 2008). These deficiencies still remain within the 
context of online examinations. 


In facing the uncertainties of such a global pandemic, educators must consider alternative 
methods to address potential problems when evaluating learning and offer required assistance in 
accordance with the skill level of the students to promote their success. Instead of relying heavily 
on conventional assessment approaches, it is imperative for educators to reimagine teaching and 
learning experiences to employ exceptional pedagogical practices to advance education to the 
next level. 


Learning Assessment during COVID-19 

Online assessment in the time of COVID-19 is an example of why educators need to integrate 
new methodological and technological strategies to provide flexibility in evaluating student 
learning outcomes, while also transforming assessments into learning experiences. 


The traditional summative assessments at this time were faced with challenges of transferring in- 
class exams into online quizzes. As students shifted to remote learning, there was a greater need 
for flexibility, understanding, trust, and compassion to keep them engaged amid the COVID-19 
pandemic. However, maintaining a more conventional approach to assessment, with limited 
schedules and means for students to access and respond to the questions, lead many students to 
struggle with taking online examinations (Fuller, et al., 2020; Tuah & Naing, 2021). 


The scores for these high-stakes tests were also criticized due to issues of validity and reliability 
in determining students' performance (Tuah & Naing, 2021). The performance-oriented 
assessments focused heavily on memorization, rather than critical thinking and overlooked 
individual student differences in learning styles, as well as their varying levels of test anxiety 
(Harsy, et at., 2020). Students' perceptions of these examinations affected their study behaviors, 
encouraged them to overemphasize the importance of exam scores, and even worse, caused 
cheating on the test. This further demonstrates the issue of academic integrity, which remains a 
worry of educational equity in remote education (Gamage, et al., 2020; Lanier, 2006). 
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The faculty at our university revealed similar concerns about using summative assessment with 
high-stakes examinations during the pandemic semester. Their feedback, in conjunction with the 
existing barriers for summative assessments, highlights a need for alternative options that 
concurrently evaluate how students are doing with remote education and provide assistance for 
individual difficulties to support their learning. Developing such an assessment strategy that truly 
reflects student achievement with an emphasis on student well-being, ensures the quality of 
assessment, as demonstrated by this case at our university during the COVID-19 pandemic. The 
working model provides flexible, humanistic, and practical approaches to enhance learning 
innovation and support the transformation of higher education in the post-pandemic era. 


This paper explores a theory-based practice derived from mastery learning that allows students to 
choose their own “adventure” and master the topics progressively with scaffolded feedback in 
the Introduction to Linear Algebra and Differential Equations course. This strategy aligns with 
the principles of empathy, flexibility, pragmatism, and simplicity to provide multiple 
opportunities and ways of learning to accommodate different student needs in preparing for the 
semester amid the pandemic. 


This experience shows a deep collaboration between experienced instructors, academic 
researchers, learning designers, and learning analytics specialists in assessing the effectiveness of 
such a mastery-based approach to instruction and assessment. Through a post-course survey and 
students' performance, we derived the academic implications of this strategy through data-driven 
results. The findings can also inform ongoing improvement of mastery-based assessment to 
ensure its effectiveness and sustainability. 


Transformation Measure to Turn Exams into Learning Experiences 


Mastery learning 

Given the evidence found by previous studies and our faculty members' feedback, traditional 
summative assessments with high-stakes tests did not align with our educational goals to support 
and enhance student learning during the pandemic. Finding an alternative approach to design a 
new assessment plan became critical. We began with a theoretical basis for the purposes of 
teaching and learning, finding a student-centered educational strategy called mastery learning to 
design the assessment activities. The idea of mastery learning is not new to Western educational 
thinking and the mastery-based approaches also have been implemented in pedagogical settings 
for many years (Block & Burns, 1976; Guskey, 2010; Kulik, Kulik & Bangert-Drowns, 1990). 
John B. Carroll (1963) initiated the conceptual model of mastery learning based on the premise 
that students can achieve the desired level of mastery in a given subject with sufficient time to 
practice. Then, Benjamin Bloom (1968) carried on and transformed this concept into a working 
model to outline a number of specific steps to achieve a distinct level of mastery. Other 
educators continued refining and elaborating on Bloom’s Learning for Mastery model to make it 
more systematic and practical (Block & Burns, 1976). 


Mastery-based assessment 

In mastery learning, the assessment attempts to address students’ deficiencies of needed concepts 
and skills through regular formative tests and provides them opportunities to develop proficiency 
and confidence to achieve the desired level of mastery on the learning topics (Block & Burns, 
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1976; Guskey, 2010; Kulik, et al., 1990). To gain such an improvement in student performance, 
a mastery-based assessment includes pre-and post-tests, formative measures, corrective 
instruction, and enrichment activities to scaffold student learning (Guskey, 2010). A well- 
designed assessment plan based on mastery learning could minimize individual differences in 
their aptitude for the subject and provide assistance to equip each student with a confident 
command of the fundamental concepts to progressively master the learning content at their own 
pace (Block & Burns, 1976). This assessment type is individually based and permits multiple 
chances for testing to cater to individual needs to support student learning. 


Self-perceptions of learning 

The essential criterion of success in mastery learning is how much improvement individual 
students display. Mastery-based assessments can mitigate the risk of test anxiety and related test 
perceptions on a student’s performance in summative measures, such as final examinations and 
other end-of-instruction tests (Block & Burns, 1976). For students with a fixed mindset, the 
failure of those performance-oriented exams infers that they are not smart enough to succeed and 
then they quickly give up (Boaler, 2013). Therefore, the use of high-stakes examinations could 
aggravate the notions of limited intelligence or fixed ability in learning. The assessment in 
mastery learning is not a one-shot or do-or-die test experience. It focuses more on the progress 
students make in learning (Block & Burns, 1976; Guskey, 2010; Kulik, et al., 1990). The 
opportunities to grow help students believe their knowledge or skills can be developed via 
learning (Boaler, 2013). 


Motivational and academic supports 

In addition to encouraging students in the learning process, mastery-based approaches to 
assessment and instruction also have motivational benefits for instructors in connection to 
increased responsibilities for students’ learning outcomes and higher expectations for students’ 
academic success. (Zimmerman & Dibenedetto, 2008). The progress students demonstrate makes 
instructors feel rewarded and are then more willing to contribute their time and effort to help 
pinpoint topics that students struggle with and provide personalized instruction to support their 
learning. Furthermore, previous studies have indicated that using mastery learning techniques 
has positive effects on student academic achievement (Block & Burns, 1976; Guskey, 2010; 
Kulik, et al., 1990). Given the mental and academic benefits of mastery learning, this 
pedagogical approach could help address the challenges we faced during the COVID-19 
pandemic in assessing students’ performance while also preparing for the change needed to 
transform higher education in the future. 


A Mastery Learning Intervention in a Mathematics Course 


At our institution, two experienced professors from the Department of Mathematics proposed to 
initiate a new assessment approach using a mastery-based grading system to replace high-stakes 
testing for their Introduction to Linear Algebra and Differential Equations course (see Appendix 
A for more detail). The original assessment of the course was mostly based on a few high-stakes 
tests, including three midterm exams (20% each) and one final exam (30%). The instructors 
attempted to replace the existing assessment model with mastery-based assessment approaches to 
enhance learning and reduce stress in students during the pandemic. They provided a series of 
low-stakes quizzes for each topic to allow students to practice and used a mastery-based grading 
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system to authentically evaluate student achievement. The design and development of this 
assessment model was fueled in the pursuit of an urgent need to ensure students were learning 
the requisite material while simultaneously maintaining student engagement, satisfaction, and 
access to quality assessment. 


The new grading system, using more frequent low-stakes formative assessments with a mastery- 
based grading strategy, allows students to evaluate their own knowledge of learning topics and 
continue to progress toward proficiency to build the required math skills and understanding. 
Faculty engage students with performance feedback and provide assistance based on the 
students’ level of understanding. This ongoing assessment and adaptation of support enabled 
both students and instructors to monitor progress and then the instructor was able to provide 
tailored feedback and scaffolds to help further learning. Scores in this model, unlike those of 
more traditional exams, serve multiple functions in communicating student performance. Scores 
provide motivation and feedback to students and guide faculty towards appropriate modifications 
to the course content and relevant instructional materials. 


The implementation of this assessment plan adhered to the common essential feature of mastery 
learning, including clear learning goals and expectations, credit only for mastery, and multiple 
low-stakes tests to progressively reach mastery (Table 1). In this course, students took 13 
quizzes, two take-home assignments, and one final reflectional quiz instead of midterm and final 
examinations. This low-stakes assessment strategy provided students with frequent opportunities 
to practice in order to develop and master the content. The professors graded student learning 
outcomes through a mastery-based grading system as opposed to a percentage grade of correct 
questions on each individual quiz. After each learning topic, students had six opportunities to 
demonstrate their understanding of the topic. They were required to achieve the desired level of 
mastery, which was at least four correct answers. Students could recognize earlier that they had a 
developing, but not proficient understanding of the objectives and the instructors offered extra 
instruction to scaffold students and then gave them additional opportunities to display mastery. 
This mastery-based grading technique was the assessment of learning objectives for each topic as 
a prerequisite for advancement. 


Table 1 
Summary of the mastery-based assessment method employed in the course 


Low-Stakes Assessment Mastery-Based Grading 


e Based on frequent assessments with each e Students had six different questions on 


counting less. each of 21 topics spread across at least 

e This course shifted to 13 quizzes, two three assessments during the semester. 
take-home assignments, and a final e They needed to get four questions right 
reflective exam to make up 84% of the to have “mastery” of the topic. 
course grade. e Students received feedback after each 

e Student effort is spread throughout the quiz and then could improve for the next 
semester. time they saw that topic. 
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Study Focus and Methodology 


To examine the impact and perceptions on student success, retention, and satisfaction of the 
mastery-based assessment approach employed during the pandemic, we compared students’ 
average GPA scores with the historical data from past course sections without a mastery-based 
approach to understand any changes in student academic achievement. We also surveyed the 
students and the instructors regarding their feedback on this course to explore the perceived 
impact of this assessment technique in order to answer the following research questions: 


e QI. What's the impact of the mastery-based grading approach with low-stakes 
assessments on student academic achievement in an undergraduate mathematics course 
during the pandemic semester? 

e Q2. What were the perceptions of students regarding the use of low-stakes assessments 
and mastery-based grading approaches? 


Results and Discussion 


This mixed-method study was conducted to collect and analyse both quantitative and qualitative 
data to evaluate the impact of mastery-based grading with low-stakes assessments on student 
learning. We employed a descriptive statistics method to summarize the differences of the 
average GPAs between this course and past course sections without mastery-based assessment. 
The results help in showing whether this new assessment model could enhance student 
achievement. The data of the student survey and instructors' feedback were analyzed by 
experienced analysts to interpret the most common and overarching themes regarding both 
students’ and instructors’ experiences with this mastery-based assessment strategy to help 
understand the effectiveness of this approach from their perspectives. The main findings 
synthesized from both qualitative and quantitative data are presented in the following section. 


Increased academic achievement 

Based on the historical achievement comparison, the average GPA for the low-stakes and 
mastery grading class was higher than the previous courses, which used traditional summative 
assessment with high-stakes tests. When comparing cumulative GPA of the pandemic semester 
with the overall GPA of the same course from past semesters, the results (figure 1) show an 
improvement in students’ academic achievement. The average GPA of fall 2020 (3.73) is 0.7 
points greater than the average GPA (3.03) of the course from the past 14 years. This result 
echoes the argument from previous studies about the academic effect of mastery-based learning 
approaches on student achievement (Bloom, 1968, Block & Burns, 1976; Guskey, 2010; Kulik, 
et al., 1990). 
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Figure 1 
Historical comparison of students' average GPAs of this course 
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Engaging learning experiences 

To round out a thorough picture of the effects that mastery-based assessment strategies have had 
on student learning, we also analyzed student survey data to report the findings in more detail. 
We received a survey response rate of 41.91% (57/136). The findings revealed that students felt 
engaged with a mastery-based assessment approach. 97% of students considered this method to 
have had a positive impact on their learning (i.e., motivation, success, interest, confidence, 
fairness). 93% found it reduced stress, prevented procrastination, decreased test anxiety, and 
increased timely feedback. 79% reported they spent the same or more effort preparing for 
informative assessments as traditional high-stakes exams. 


In addition, students studied more regularly in the semester and learned from their mistakes. For 
example, a student described this as follows: “Getting questions wrong really is a chance to 
learn and improve, not just watch your grade drop.” The instructors also found students were 
more engaged and came to office hours more often and asked more questions outside of class. 
These findings were correlated to students' perceived usefulness of these assessment activities in 
the course in supporting their academic growth. 


Implications (A Look at the Future) 


The university was devoted to preventing coronavirus-related disruption for instructional 
continuity. The rapid shift to remote education challenged educators to maintain effective 
instruction using conventional pedagogical practices during the pandemic semester. For example, 
traditional summative assessment with high-stakes tests were criticized in regard to validity and 
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reliability in assessing learning performance while also increasing anxiety in learning, which hurt 
student mental health. 


Given consideration of the issues with a traditional assessment model, the instructors adopted a 
mastery learning framework to redesign the assessment plan for the Introduction to Linear 
Algebra and Differential Equations course. They utilized (a) formative low-stakes assessments, 
(b) a mastery grading framework, (c) administered using take-home and proctored assignments, 
(d) and longer cumulative opportunities to create a flexible assessment model. The flexible 
assessment model asks students to master or fully understand the concept of assigned topics at 
their own pace before moving forward. This learner-centered practice opens up personalized 
learning paths and pace and allows educators to better scaffold individual needs to build teaching 
resilience amid a pandemic. 


This paper discussed how this new pedagogical practice was developed and delivered to improve 
the flexibility in assessment. The follow-up study conducted by the faculty and learning 
specialists from the teaching and learning institute, indicated that this flexible assessment model 
has an impact on student success, and can inspire alternative pedagogical strategies to advance 
teaching and learning. We believe what we have done cannot just help the university to continue 
educating during the pandemic, but also promote learning innovation that supports the vision for 
the future of higher education. 
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Appendix A 


The Syllabus: Assignments and Grading Policies 
This course uses a grading system developed based on Standards Based Grading. This will be 
graded in a way that is significantly different from other math classes you may have taken in 
the past. We have identified 21 skill sets (or standards) that we wish you to master by the end 
of the course. Standards based grading is that you just have to show mastery of these skills to 
get an A. This will be evaluated with weekly quizzes during TUTORIAL and a timed take-home 
midterm and a final examination. You will not receive credit as a percent grade. Rather each 
question will be evaluating a particular skill set. Each question on the quiz will be clearly 
marked which standard (skill) is being tested. If you do the problem and use the skill correctly 
then you get a point in that skill. (For fans of video games, think of it as levelling up your 
character and you need to level up each skill of that character). In order to show mastery of a 
skill, you will need to demonstrate successful use of the skill 4 times. You will get at least 6 
opportunities to demonstrate each individual skill. The final will test each skill at least once. It 
is your final opportunity to demonstrate mastery. 


There are 21 standards (skills) each worth up to 4 points and the homework is worth a 
total of 16 points. 
e 88-100 pts A 
67-87 pts B 
47-66 pts C 
26-46 pts D 
Below 25 F 


+/- will be determined by the end of the semester. That means if you have a 90, that 
could be an A or an A- depending on the final. 

** Note that failure to do homework and attend class can lower your final grade by as 
much as a full letter grade. 


The Skills you will be acquiring in this course are as follows: 

Use Gaussian Elimination to set up problem and express solution in parametric form 

Be able to add, multiply, transpose matrices 

Decide something is or is not a Linear Transformation and be able to construct Standard 
Matrix 

Understanding Linear Combinations and Linear Independence 


SE NOS 
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Basis of Column Space & Basis of Null Space 

Coordinates and the Change of Basis Matrix 

Identify Vector Space and it's subspaces and then find its basis and dimension 
Abstract Vector Space is or is not Linear Transformation 


. Find Coordinates for Abstract Vector Spaces and find basis for Kernel and Range. 
. Compute determinant and its properties. 

. Finding eigenvalue and basis for eigenspace. 

. Determine Diagonalizability 

. Use Gram-Schmidt to compute projections using orthogonal basis 

. Compute Least Square Solutions and use least square solutions to compute projections 
. Checking solutions and checking autonomous solutions 

. Solve Separable Equations and First Order Linear 

. Solve Exact and Almost-Exact 

. Be able to model systems with DifEq 

. Solve 2nd Order Constant Coefficient Homogeneous Differential Equations 

. Solve 2nd Order Non-Constant Coefficient Homogeneous Differential Equations 

. Solve 2nd Order Nonhomogeneous Differential Equations 


424 


Pep Rally Using One-on-One Faculty Consultations 
to Promote Technological Pedagogical Content Knowledge 


Natalia Kavun, PhD, 
Fordham University 
416 E 81" St., apt. 4D, 
New York, NY, 10028 


Index words: TPACK, faculty consultations 


Abstract 

This paper provides explanation of the use of the Technological Pedagogical Content 
Knowledge framework (TPACK) and its role in the one-on-one faculty on the subject of 
instructional design (ID). Additionally, it explains the application of this theoretical framework 
to the faculty who are at the beginning stages of their consultations with instructional designers. 
Thus, such faculty has different expectations, and it is important to gauge their current 
pedagogical practices to successfully assist them. The peri-pandemic ID team applied such an 
approach through their experiences with faculty in the Business School. As a result, one-on-one 
consultations became a key solution for the relationship and course development with the 
faculty. 

Introduction 

The field of instructional design (ID) is a relatively new domain. However, the COVID 
pandemic emphasized the need for ID experts in higher educational institutions as the demand 
increased significantly due to a prompt transition to online learning in Spring 2020. The urge for 
creation of such departments have been greatly noticeable across various institutions. O’Keefe et 
al. (2020) stated that faculty did not have enough time to gradually transition their face-to-face 
classes to online modalities and in this case, the role and help of IDs was reiterated again. 

Therefore, a new term Emergency Remote Teaching has been framed to properly refer to 
such a rapid transition (Hodges et al., 2020). The knowledge that instructional designers bring 
into the institutions is not limited by technological aspect of it. They are experts in the 
curriculum development, course design, and creation of training programs. However, there is still 
a lack of understanding of differentiation between instructional technologists and instructional 
designers due to the fact that instructional designers possess both a toolkit and pedagogical 
knowledge that can bridge the gap in the pedagogical practices of the instructors with the 
incorporation of instructional technology in their courses. Therefore, it is important to be in a 
close collaboration with the subject matter expert (SME), but to never cross the line of defining 
the subject content of the course (Halupa, 2019). Thus, ID Teams play a crucial role in course 
design, but such teams should always have a strategy that they follow in order to successfully 
implement their services at the institutional level. 

Needless to say, that the duties of new instructional design teams are not well defined as 
such new departments are yet on the way of determining their role and significance at the 
institutions. Faculty may not always know what to expect from such collaboration and what 
resources instructional designers have to offer. Therefore, there is a struggle of establishing an 
effective collaborative approach with faculty that would benefit the faculty and students’ 
teaching and learning experiences in the long run both pedagogically and technologically. 

One-on-One Faculty Consultations 
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This paper focuses the development process and workflow of a peri-pandemic 
instructional design team in the business school setting and establishment of its collaboration 
with faculty. It provides insights to the important steps of ID team building and the approaches to 
one-on-one online faculty consultations since the launch of the new ID team was provoked by 
the pandemic. Thus, immediate steps were taken to engage faculty and facilitate their distance 
learning practices by initiating direct contact with the SMEs. Throughout the process of team 
development, instructional designers recognized that faculty members are at different levels of 
technology integration in their teaching which has an impact on their overall adoption of 
classroom technology and its use for online and hy-flex learning experiences. Soto & Smith 
(2020) define hy-flex as a model of instruction that allows both teaching in-person and online 
simultaneously. As a result, one-on-one consultations were offered by instructional designers in 
order to meet individual pedagogical needs of faculty members regardless of the mode of 
instruction or level of technology adaptation. It was important to ensure that all the faculty 
members are able to teach in the preferred modality. Thus, the team was able to promote 
instructional technology integration to complement the pedagogical practices and content 
knowledge of faculty members depending on their needs and teaching expectations. 

Additionally, one-on-one consultations allowed instructional designers to build 
trustworthy relationships with faculty from scratch. Such close contact has been noted to be 
beneficial since IDs were able to cater to the individual needs of SMEs which vary across the 
disciplines. The team had to overcome the challenges associated with only online 
communication, considering the fact that IDs have not met faculty in person due to pandemic 
circumstances. However, the process was more complex since the team existed fully online and 
faculty members were not acquainted with the advantages of such collaboration. The team lead 
emphasized the need of notetaking regarding the faculty interactions which further led to the 
classification of the faculty based on their requests and level of technology use. In such a way, 
certain patterns and outcomes of one-on-one consultations were identified which provided a 
clearer direction for the IDs on how they can develop their relationships with faculty and take it 
beyond technology consultations which was still very beneficial for the faculty who were mostly 
holding synchronous online classes at the moment. All this work has been done considering the 
future work of the team. Faculty classification based on their technology adaptation level in 
online learning informed IDs regarding the trainings that meet the existing needs of faculty both 
in pedagogical and technological domains. 


TPACK Framework 

The active approach of the analysis that the team has taken led to understanding that it 
falls under the Technological Pedagogical Content Knowledge (TPACK) framework (Koehler & 
Mishra, 2013) since IDs consider the technological readiness of SMEs and makes suggestions for 
the course design and activities. It is important to take it one step at a time and not overwhelm 
the faculty with the content throughout the consultations, especially when all the instruction took 
place online. Therefore, TPACK provides more clarity on the elements that IDs consider while 
conducting the consultation: technology readiness, pedagogical component, and subject content. 

TPACK framework explains the interconnection of three knowledge forms: 
Technological knowledge (TK), Pedagogical Knowledge (PK), and Content Knowledge (CK). 
This theory looks into the overlaps that such primary knowledge can create, and these overlaps 
are viewed as the most valuable practices in instruction: Pedagogical Content Knowledge (PCK), 
Technological Content Knowledge (TCK), Technological Pedagogical Knowledge (TPK) 
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(Koehler & Mishra, 2013). It is important to understand at which level of TK every faculty 
member is and how an instructional designer can assist to incorporate the knowledge to 
potentially bring the course to the level at which all of the three primary pieces of knowledge 
overlap and create an ideal environment to improve teaching experiences for instructors and 
learning for students. 


Figure 1 
Visualization of TPACK 


Technological 
Pedagogical Content 
Knowledge 
(TPACK) 


Technological 
Pedagogical 
Knowledge 
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Technological 
Content 
Knowledge 
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Pedagogical 
Content 
Knowledge 
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Contexts 


Note. Figure | shows the representation of TPACK framework by mkoehler. Reproduced by 
permission of the publisher, © 2012 by tpack.org (Mkoehler, 2011). 


Thus, TPACK is often applied in professional developments for teacher education. Such 
complex relation between three types of knowledge emphasizes the importance of the relations 
that lay in between. In such a way, instructors are able to promptly adapt to the needs of the 
course and students’ alterations. Mishra & Koehler (2012) explain that this model is an extension 
of Shulman’s idea of Pedagogical Content Knowledge (1987). Thus, the current state of 
education requires instructors to leverage various knowledge types to promote the best practices 
of student learning even during the worldwide pandemic. 


Three Consultation Approaches 

Based on the TPACK framework, Koh (2020) identified three approaches that 
instructional designers implement during their one-on-one faculty consultations with faculty 
members: technology modeling for faculty without significant TK, pedagogical realignment for 
faculty with some TK knowledge, and deepening practice for faculty with vast TK. Likewise, the 
described instructional designer team has created a database of faculty that they worked with and 
was able to identify certain patterns in the nature of one-on-one consultations that adhere to 
Koh’s approaches of implementation (2020). Based on the initial requests and consultations, IDs 
are able to identify which strength the SME possesses and build upon that skill in order not to 
overwhelm the faculty member with a lot of new information. 
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Technology modeling approach is used with instructors who use a very minimal amount 
of technology in their classes (Koh, 2020). One of the primary reasons for such behavior is the 
lack of experience and in this case, one-on-one sessions are a great opportunity to give a short 
demo for the potential application of the tool in the course design. Moreover, Koh identified that 
when the instructional design team carried several faculty workshops, it increased an interest in 
the work of instructional designers and improved collaborative practices. Likewise, a positive 
dynamic has been experienced by the team when the decision was made to take a gradual 
instructional design immersion with faculty who choose to stick to more basic online course 
design with the limited toolkit usage. It is important for the course designer to be able to preview 
the course before the one-on-one consultation in order to know which primary knowledge in the 
TPACK framework may be prevalent and which overlaps already exist. In our case, IDs noted 
that one-on-one demos of the use of the certain technology may not always be enough. 
Oftentimes, ID create recorded instructional videos addressing a specific faculty question or 
detailed instructions that are provided to the instructor, so they are able to take more control over 
the reoccurring issue. For instance, instructors could make a request to demonstrate how to use 
the university LMS for assessment purposes. Additionally, some basic technological requests 
included identifying location of recorded class sessions. 

Pedagogical realignment improves the PCK area of TPACK framework (Koh, 2020). 
According to the author, such an approach is taken towards the faculty who are willing to try out 
new approaches and technology. In such consultations, the role of an instructional designer is to 
ensure that the learning takes a student-centered standpoint. Also, the learning is engaging in the 
classroom environment using interactive tools for formative assessments and incorporation of 
group work. If students are expected to submit a multimedia assignment, instructor is advised to 
provide clear guidelines. In this case, ID Team assisted more significantly with pedagogical 
content, once the technological needs were fulfilled, specifically for online courses. It was more 
important for the instructor to be able to navigate technology for the classroom purposes and be 
confident in using it. After that need was met, IDs suggested to assist in creating more various 
content for online classes, such as self-paced instructional videos that incorporated interactions. 
Additionally, IDs would provide more ideas on how faculty can vary online class activities in 
order to increase student engagement throughout the sessions. 

Deepening practice is applied to the faculty with more advanced experiences in all the 
knowledge areas. Such faculty were offered additional options to incorporate more robust 
instructional designs in their teaching with technological integration. Such cooperation enabled 
some of the faculty to have hybrid-flex classes which allows to mix different teaching modalities 
and audiences (Beatty, 2019). During Spring 2020, the instructional design Team Lead in 
collaboration with the Information Technology Department created a physical space that can be 
used both for students who are taking a class face-to-face and virtually. Additionally, it is worth 
mentioning that Business School has students who are located on campuses overseas. Hybrid- 
flex modality allowed to combine students who are able to learn in person and who can only 
come to class through the virtual conferencing platform. One-on-one consultations help to create 
deep reflective sessions on the current teaching practices that help instructors to come to a 
potential solution through the series of questions. Additionally, instructors who incorporate 
deepening practices in their courses are invited to share their experiences with the peers 
throughout the workshop series. It is an opportunity to share and reflect on the ongoing practices. 
Currently, deepening practices allow IDs to revamp the courses for specific faculty. It is worth 


428 


mentioning that adjunct faculty members are interested in ID help and they often utilize the team 
to bring new learning experiences to their students. 


Benefits of One-on-One Consultations 

The approach using three different types of consultations with faculty is based on 
TPACK and previous research. The ID Team identified a number of advantages that cause 
positive change in the collaboration with faculty. Such individualized approach allows IDs to 
create long-term collaborations with faculty that may go through all three stages of the 
consultations depending on the current concern of the faculty. 

Instructors are able to improve their course materials with the help of technology. 
Oftentimes, they may not be aware of the possibilities and resources that are available to them. 
IDs work closely with faculty and point out technology that may assist in the course design 
content. Undoubtedly, some of the faculty do not prefer learning new technology. However, after 
ID demonstrated the improvement of course content and provides some initial trainings using the 
tool, such approach sparks further interest in collaboration and acquisition of new technology. 
Thus, IDs are often rewarded as faculty are satisfied with the overall improvement of the class 
materials or activities. 

It can be challenging to work with the faculty on alternating their pedagogical approach. 
Due to emergency teaching practices, instructors did not have time to readjust their face-to-face 
courses and transform them into online sessions. Therefore, IDs were making attempts to help in 
the pedagogical realignment process by sharing the insights of the other faculty members with 
their assigned instructors. This ID Team takes advantage of the existing practices at school that 
already meet the needs and the goals of the departments. 

More importantly, exposure to new technology often realigns the pedagogical strategies 
that are used by the instructors. Understanding that there are more opportunities to diversify the 
content enables instructors to consider active learning activities in their classes. It is beneficial 
since it allows students to be exposed to a variety of activities that are still beneficial for their 
learning practices. A lot of professors were able to adapt their novel emergency teaching for 
online environments in the current face-to-face classes, as well. 

Lastly, many instructors considered updating their class materials, including PowerPoint 
presentations because ID Team led several workshops guided towards multimedia design. 
Instructors became more interested in making their slides more appealing to the students. 
Needless to say, that with such a rapid development of multimedia and technology, students have 
high expectations of the content that they are exposed to. Therefore, technological modeling 
approach plays an important role in the content improvement. 


Disadvantages of One-on-One Consultations 

From the overall experience, the ID team noted that it can be challenging to move away 
from the technological consultations and be perceived as a team that can contribute to the 
pedagogical domain, as well. Emphasizing the use of technology may disregard the focus on the 
pedagogical approach and it can be hard to achieve the overlap in TPK component of TRACK 
which leaves TK as a stand-alone knowledge that is not appropriately incorporated into the 
course design from the pedagogical standpoint. In this case, technology is used to its minimum 
potential. 

Another concern that the new ID Team encountered is the lack of identity of the ID 
experts. In some cases, the team is mistaken for IT department. In this case, the talent of the IDs 
may not be used to the full potential. Expertise and assistance with technological issues misleads 
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faculty and the communication becomes rather transactional, and it is limited to TK issues. It is 
important to have events that show the full potential of the team. In such cases, faculty gets a 
better understanding of the overall purpose of the team. As mentioned before, this ID team was 
created during the pandemic, and it is one of the reasons why there are some misconceptions 
regarding the purpose of the team. 

Taking such an approach required better cooperation amongst the IDs to ensure that they 
provide services at the same level. The goal was to ensure that instructors have the same 
experience regardless of the ID that they closely work with. In order to quality check and 
synchronize the work of IDs, the ID Team was keeping records of the collaborations with 
instructors and discussing as a team each collaboration to ensure that instructors receive an 
appropriate consultation from each member of the team. This process is time consuming, but it 
allows IDs to provide peer feedback and identify the core values of the team that serve purpose 
to the faculty. 


Conclusion 

In conclusion, the use of TPACK in one-on-one faculty consultations help to improve the 
building of long-term relationships with instructors as the ID team takes them through all the 
areas to reach the final goal of the improved course design. It is quite challenging not to fall into 
the trap of focusing too much on TK during the pandemic as it was a necessary measure. 
However, these tendencies have been changing since a lot of instructors went back to face-to- 
face teaching and now PK becomes more of the priority. It is important to state that the pandemic 
itself has taken toll on IDs, instructors, and students. Thus, one-on-one consultations became a 
helpful way of creating a safe space for interactions that lead to fruitful collaborations. The ID 
team anticipates more improvements in their work with faculty as the transformations of their 
work together are still happening. Keeping track and records of the consultations allows IDs to 
reflect on their practices and ensure that they are successfully working towards the overall course 
design improvement as demanded by faculty needs. 
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Abstract 


The COVID-19 pandemic has forced K-12 schools to seek professional development with online 
resources that address their professional needs while keeping health and safety in mind. In 
Missouri, approximately 250 school districts in the Network for Educator Effectiveness (NEE) 
have 24/7 access to the EdHub Library that provides more than 500 online self-paced 
professional development resources (Network for Educator Effectiveness, n.d.-a). Due to the 
pandemic, the EdHub Library has seen increased web traffic; this study identifies the most 
accessed resources by analyzing user and server web analytics to improve resource access. This 
study also identifies resource access patterns using the Apriori algorithm for association rule 
mining. The results pointed out that approximately 40% of the web traffic occurred among the 
EdHub Library homepage, Dyslexia activities, and resource page for Teaching Standard 1 (i.e., 
Content Knowledge and Appropriate Instruction). Along with the findings, recommendations for 
resource optimization by the Word Wide Web Consortium (W3C) are explored. The 
visualizations can be found on Tableau Public (Leung, n.d.). 


Keywords: Teacher Professional Development, Educational Data Mining, COVID-19 
Introduction 


As the World Health Organization (WHO) declared the novel coronavirus (COVID-19) 
outbreak a global pandemic on March 11th, 2020, this study reports on the evaluation efforts for 
optimizing online resources for teacher professional development (World Health Organization, 
2020). Since the beginning of the pandemic, 1.3 billion learners are still affected by school and 
university closures as institutions implement entirely online and hybrid solutions (UNESCO, 
2020). As K-12 educators adapt their face-to-face curricula to distance and hybrid formats, the 
professional development needs of teachers have been neglected since the beginning of the 
pandemic. All in-person teacher professional development (PD) opportunities came to a halt, 
whereas fully online PD provided teachers access to online self-paced resources and synchronous 
webinars through web conferencing tools (e.g., Zoom). 


In Missouri, the EdHub Library is an online professional development platform for K-12 


teachers and school administrators as part of the Network for Educator Effectiveness (NEE), 
which is a comprehensive education assessment system (Network for Educator Effectiveness 
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n.d.). While the EdHub Library provides teachers with 24/7 access to over 500 online resources, 
the platform has experienced increased web traffic since the beginning of the pandemic. The 
purpose of the study is to evaluate and improve access to online resources by analyzing user and 
server web analytics requests. 


Literature Review 


The following literature review describes the components of teacher professional 
development and recommendations for effective implementation. Research studies support 
positive outcomes and perceptions of online teacher professional development. 


Understanding the Needs of Teacher Professional Development 


Research studies have been conducted to examine the components of teacher professional 
development. Desimone (2011) described the core features of effective teacher PD in terms of 
(a) content focus, (b) active learning, (c) coherence, (d) duration, and (e) collective participation. 
Garet et al. (2001) examined the relationship between the characteristics of teacher professional 
development and teachers’ self-reported outcomes. Three recommendations emerged from 
integrating effective professional development and self-reported outcomes: form, duration, and 
participation. Form refers to how PD activities are structured either as a “form” or “reform” type 
of activity. The first type of activity describes traditional workshops as episodic events, whereas 
“reform” activities are collaborative workshops. The second recommendation is the duration of 
these activities. When teachers are expected to participate in longer sessions, follow-up and 
continuous support are essential to effective PD. The third recommendation is participation. The 
researchers recommended that teachers and support staff be included in the activities that 
promote everyday teaching tasks. Garet et al. (2001) findings agree with similar studies on 
effective PD delivery in terms of (a) sustained time, (b) collaborative and active participation, (c) 
content-driven, and (d) situated activity (Archibald et al., 2011; Borko, 2004; Darling-Hammond 
& Bransford, 2006; Desimone et al., 2002; Elmore, 2002; Guskey & Huberman, 1995). 


Effects and Perceptions of Online K-12 Teacher Professional Development 


Yoo (2016) examined the effects of online professional development on 148 teachers’ 
self-efficacy twice with a five-week gap after completing a learning module. In the analysis of 
the survey results, teachers reported professional enhancements through goal setting and concrete 
learning strategies, adjustments of their frame of reference in terms of their evaluation and 
understanding of a certain level of helplessness, and uncertainty in dealing with student and 
school factors, such as aptitude scores and curriculum guidelines. Whitaker et al. (2007) studied 
the effectiveness of an online platform called MyTeachingPartner (MTP) using web server logs, 
teacher evaluation survey responses, and focus groups on teachers’ beliefs. The researchers 
reported that teachers looked for practical activities by understanding the needs of their students. 
Although the MTP materials were useful, teachers said that materials needed to be practical for 
their teaching. Rice and Dawley (2009) analyzed survey responses of 259 teachers using the 
online K-12 platform called “Going Virtual!” to understand the practices and models of online 
teacher PD. Despite well-established national guidelines, school administrators were aware of 
these guidelines but implemented procedures based on their teachers’ needs. 
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The Need for Evaluation in oTPD 


Studies have explored teacher professional development as a critical factor that impacts 
student achievement (Campbell et al., 2004; Darling-Hammond & Bransford, 2007). With the 
growth of educational technologies and teachers having less time for professional development, 
the rise of online teacher professional development (oTPD) allows for professional engagement 
in continuing education systems. Numerous research studies examined face-to-face teacher 
professional development. However, a few studies on oT PD examine online professional 
development programming and technical components behind educational systems. 


Problem Statement 


Dede et al. (2009) identified the lack of empirical evidence on effective online teacher 
professional development design. Current research on oTPD examines four aspects in terms of 
(1) program design, (2) effectiveness, (3) technical design, and (4) learner interactions. Program 
design refers to the evaluation of content and best practices in methods of delivery. Program 
effectiveness explores the short-term outcomes provided by oTDP, which are generally teachers’ 
self-reports. Program technical design evaluates the effect of communication tools on specific 
goals. Learner interactions refer to the quality of participation in online communication and 
collaboration. While present studies explore the program design and effectiveness aspects of 
online teacher professional development, this study evaluates the use and access patterns of 
resources with web analytics for assessing the technical and learner interactions aspects of oTPD. 


Purpose & Significance of the Study 


This study seeks to identify the most accessed resources and develop user and server 
access patterns in online teacher professional development resources. The significance of this 
study involves providing improved access to resources, especially to rural regions in Missouri 
school districts. The study explores the following research questions: 


@ RQ1: Which resources do users access the most based on Page Depth and Time On 
Page? 

@ RQ2: Which resources have the highest Server Response Time, Document Content 
Loaded Time, and Page Loaded Time? 

@e RQ3: What access patterns exist for users and server document requests? 


Methods 


This study uses exploratory analysis of 164,772 web resource transaction records from 
May 2018 - May 2021 using user and server web analytics metrics to identify the most accessed 
resources and develop navigation patterns using association rule mining. First, the study explores 
the most accessed resources using Google Analytics user metrics, including Page Depth and 
Time On Page. According to Google Developers, Page Depth refers to the average number of 
pages users visited within a 30-minute session. Time On Page is calculated by the time 
difference between the user’s starting point on a particular resource and when the user moves on 
to the following resource (UA Dimensions & Metrics Explorer, n.d.). The study also uses Google 
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Analytics server metrics, including Server Response Time, Document Loaded Time, and Page 
Loaded Time to identify the most requested resources from the web server. To investigate 
resources that need to be optimized, Server Response Time, Document Loaded Time, and Page 
Loaded Time allow identifying specific elements (e.g., video, animation, text, scripts, and 
documents) in web resources that are highly requested in the server and loaded onto users’ 
browsers. Server Response Time refers to the total time that the server takes to respond to the 
user’s request. Document Content Loaded Time describes the total time that the user’s browser 
and server take to render the documents with their respective style sheets and scripts. Page 
Loaded Time is the total time that it takes to render the whole resource page. To further explore 
user and server metrics, visualizations are published on Tableau Public (Leung, n.d.). 


Second, the Apriori algorithm is a popular method for association rule mining that allows 
establishing patterns between antecedent (1.e., if) and consequent (i.e., then) components of 
frequent user and server resource access based on support, confidence, and lift measures 
(Harikumar & Dilipkumar, 2016). The support measure is the percentage in which web resource 
transactions contain a given web resource access combination. The confidence measure looks at 
the conditional probability in which a web resource is accessed. The lift measure describes the 
likelihood of web resource transactions occurring in pairs. For this particular study, the lift and 
support measures will be examined to determine the most accessed resources by users and 
requested server resources (e.g., HTML, CSS, images, videos, and PDFs) from the hosting 
server. Average user and server metrics are reported in milliseconds (ms). Because association 
rule mining is a computationally expensive method, the analysis considers all user types, 
including new and returning users. 


Results & Discussion 
RQ 1: Which resources do users access the most based on Page Depth and Time On Page? 


In terms of Page Depth, the most accessed resources by users were related to exemplary 
teaching practices videos and classroom observation practice scenarios. The search engine 
function was also used to look for content associated with Teaching Standard 1.1 (i.e., Content 
Knowledge and Academic Language) and its scoring video examples, as shown in Figure 1. The 
most accessed content regarding Time On Page was the instructor-led principal calibration 
training and building instructional skills modules. Also, the most searched content included 
research-based practices, professional development processes, technology integration, and data- 
driven decision-making, as shown in Figure 2. 
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Figure 1 


Average User Requests by Page Depth 
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Figure 2 


Average User Requests by Time On Page 
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RQ 2: Which resources have the highest Server Response Time, Document Content Loaded 


Time, and Page Loaded Time? 


In observing Google Analytics server metrics, curriculum implementation, word 
problems, kindergarten, math problem-solving, technology integration, and physical education 
were the most solicited search functions for Server Response Time, Document Content Loaded 
Time, and Page Loaded Time, as shown in Figures 3, 4, and 5. 
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Figure 3 


Average Server Requests by Server Response Time 
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Figure 4 


Average Server Requests by Document Loaded Time 
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Figure 5 
Average Server Requests by Page Loaded Time 
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RQ 3: What access patterns exist for users and server document requests? 


User and server access patterns to online professional resources and search functions can 
be established by lift measure or the probability in which web resources are accessed together, 
and support measure or the percentage in which web resource transactions occurred in a specific 
combination. 


User Web Analytics Metrics 


In examining user web analytics metrics, specific patterns for Page Depth and Time On 
Page were found for all four Dyslexia activities sorted by the highest lift measures. In terms of 
the support measure, all four Dyslexia activities and the EdHub Library with a sitemap listing all 
resources related to Teaching Standard | (1i.e., Content Knowledge and Appropriate Instruction) 
showed around 40% of users performed the given access patterns, as shown in Table | and 2. 


Table 1 


User Average Page Depth 


Dyslexia literacy activity 2 0.404 0.983 


0.973 


0.900 


EdHub Library homepage | Dyslexia module page 0.486 0.534 
EdHub Library homepage | Sitemap Standard 1 0.401 0.440 


Server Web Analytics Metrics 


Even though the highest lift measures could be found between the EdHub Library 
homepage and the dedicated listings of online resources, the support measures for Server 
Response Time and Page Loaded Time were around 1% of access to the resources as mentioned 
earlier, as shown in Table 3 and 5. In terms of Document Loaded Time, 40% of the navigation 
access could be found among the EdHub Library homepage, Standard 1, and Dyslexia module, 
as shown in Table 4. 
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Table 3 


Server Average Response Time 


Table 4 


Server Average Document Loaded Time 


Sitemap Standard 1 Dyslexia module page 0.405 


EdHub Library homepage | Dyslexia module page 0.406 


Table 5 


Server Avera e Loaded Time 


Discussion 


For user requests, the most accessed resources were (1) the dedicated page related to 
Standard 1.1 (Content Knowledge and Academic Language), (2) principal scoring practice 
videos, (3) teaching best practices videos, and (4) walkthrough examples of classroom 
observation data collection videos. For server requests, the most requested resources were (1) 
search queries related to curriculum implementation, student motivation, kindergarten materials 
and word problems, (2) collaboration and technology, (3) NEE implementation, and (4) 
integration of technology tools. 
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Based on association rule mining, support and lift measures indicated approximately 40% 
of web resource transactions occurred on the EdHub Library homepage, sitemaps for Standard 1, 
and Dyslexia activities. Because of my current responsibilities as an Instructional Designer who 
has maintained and published online materials since 2014, I take into consideration seven best 
practices that allow resources to be accessible, including (1) avoiding unnecessary redirects of 
resources, (2) reducing the size of files, (3) designing for a variety of interaction methods 
(mobile and desktop) and internet speeds, (4) using adequate text size with minimal Cascading 
Style Sheets (CSS), (5) implementing high-quality graphics with a smaller footprint, (6) using 
cookies when necessary, and (7) reducing the size of HTML pages in terms the Document Object 
Model (DOM) to avoid rendering errors (Mobile Web Application Best Practices, n.d.). 


Video materials from teaching best practices and walkthrough examples of classroom 
observation videos are heavily accessed by users. Based on the frequent requests found in server 
metrics, video materials can be optimized for different screen sizes. While video materials are 
already optimized for the web, a recommendation would be presenting three versions of the 
video for mobile, tablet, and desktop that accommodate different internet speeds and screens. 
The ideal scenario would be hosting video materials in a streaming server that automatically 
adjusts video quality based on the user’s internet speed and device. Due to the high cost of 
streaming services and an extensive collection of video materials, these videos use progressive 
download to transfer digital media between the server and client before the user can render the 
full video. Because the length of the videos is on average less than 10 minutes, the progressive 
download occurs quickly. 


Implications & Future Research 


Based on user and web metrics, users performed generic keyword searches using the 
search engine from the EdHub Library homepage (e.g., “Teaching Standard 1,” “real world,” and 
“word problems”). It is also unknown if users were successful or unsuccessful in their searches 
based on the current dataset. The future direction of this study is to examine the users’ keyword 
searches further from the search engine service, including the queries or keywords performed by 
users, the number of searches performed, unique searches, results presented to users, and 
clickthrough rates. Based on the user behavior when using the search engine, a recommendation 
would be generating curated lists of professional development materials for high-frequency 
search terms. 


Conclusion 


This study establishes navigation patterns using user and server web analytics data from 
250 school districts in the Network for Educator Effectiveness. Based on association rule mining, 
almost half of the web traffic is attributed to the EdHub Library homepage, Dyslexia module, 
and Teaching Standard 1. After analyzing 164,772 web analytics transactions from May 2018 - 
May 2021, the study provides internal stakeholders with recommendations for improving generic 
keyword searches with curated lists of resources and optimizing current multimedia resources to 
reduce load times. 
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Abstract 


This case study (1) describes how a face-to-face university maker education course 
employing a novel instructional design was redesigned to accommodate COVID-19 safety 
guidelines, and (2) presents an exploratory case study of how affordances of online learning were 
leveraged in the redesign impacted students! experience in the course. Online learning is a timely 
topic for educators and instructional designers to consider as the world grapples with options for 
maintaining educational standards during a global pandemic (Rapanta et al., 2020). Specifically, 
this study examined the following research questions: (1) How did students describe their 
experience of taking the course during Fall 2020 with the backdrop of COVID-19 and the impact 
of stressors that accompanied the pandemic? and (2) How did students reflect on their experience 
with maker-centered learning in an online learning environment versus an in-person 
environment? 


Introduction 


In March 2020, the home university of this case closed its doors to students and faculty 
and moved teaching and learning to an emergency online format due to concerns about the rapid 
transmission of COVID-19. As the pandemic continued, the university, a large urban institution 
in the Southeastern United States, physically reopened in Fall 2020 but continued with guidelines 
to limit the size of groups that met on campus. These guidelines made it necessary for university 
course designers to quickly develop blended or online courses to meet instructional goals. 

The designers of this maker education course faced a unique challenge with this 
conversion, as the course normally employs a lab-based, novel instructional design based on a 
mastery orientation and flexibility in which students complete modules largely of their own 
choice at their own pace both in the classroom laboratory and outside of class (Cohen et al., 
2019; Cohen et al., 2020). The redesigned course employed a personalized online format that 
included whole group synchronous online meetings, individual online conferencing and in- 
person lab work, and a learning management system (LMS) that provided open access to course 
materials. 

The emergency shift to blended learning because of the COVID-19 outbreak and 
resulting school closure required the course designers to consider how online affordances could 
be used to support maker-based education. Decentering the classroom and fostering autonomy 
among learners is an essential tenet of maker-centered learning (Cohen et al., 2019). Design 
decisions were made to engender student autonomy and foster personalization, staying away 
from the typical approach of a teacher-centered classroom that relies heavily on lectures (Barab 
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et al., 2001). Course designers embedded interactive learning activities and open communication 
channels within the whole group synchronous online meetings. Giving students a forum for 
sharing their work is another principle central to maker-based learning and was another key 
consideration as course designers shifted to an online learning environment (Reigeluth et al., 
2017). In this shift to online learning, the course designers were deliberate in maintaining a 
learner-centered environment incorporating constructionist principles, empowering students to 
create tangible objects and share the experience of making in a collaborative environment 
(Cohen et al., 2019). 

To understand how this modality shift impacted students’ experiences in the course, this 
study explored the following research questions: (1) How did students describe their experience 
of taking the course during Fall 2020 with the backdrop of COVID-19 and the impact of 
stressors that accompanied the pandemic? and (2) How did students reflect on their experience 
with maker-centered learning in an online learning environment versus an in-person 
environment? 


Literature Review 


During the emergency response to COVID-19, most colleges and universities switched to 
an online format to continue coursework that began in person. Studies of students have suggested 
that academic communities experienced a unique constellation of stressors that accompanied 
both the pandemic and the academic response to the pandemic. 

Students were reportedly less satisfied with courses during the COVID-19 emergency 
response, where most face-to-face courses were moved to an online format. A study of 86 
freshman college students who attended classes at an urban university similar to the university 
discussed in this paper had 63% of their student respondents report that their experience with 
instruction was worse during the pandemic after the shift to online education (Bono et al. 2020). 
This survey, which began as a longitudinal investigation and shifted to allow for investigation 
into COVID-19 related inquiry, included two points of collection, the first collection done in 
January-March 2020, prior to the COVID-19 pandemic response and the second collection done 
March-May 2020, after courses went online because of COVID-19. Another survey of a random 
national sample of 1008 undergraduate college students developed by Digital Promise and 
Langer Research Associates indicated that 51 percent of students were very satisfied with their 
face-to-face courses, but only 19 percent of those students were very satisfied with their courses 
when they moved online in response to COVID-19 closures (Means, 2020). The participants in 
this survey began classes in person and shifted online as a response to the pandemic. There was 
one point of collection for this survey between May and June of 2020 when students were 
attending online courses due to the COVID-19 shutdown protocols enacted by their universities. 
Only 17 percent of respondents who reported being very satisfied with their classes before 
shifting online reported being very satisfied with how well they learned after COVID-19 (Means, 
2020). 

Aucejo et al. (2020) reported on the results of a survey that collected student experience 
and expectation data about the impact of COVID-19. This study collected responses from around 
1500 undergraduate students at a large public university in the United States. Data from this 
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survey indicate that learning during COVID-19 changed student expectations for graduating, 
with more than 13% of students reporting that they will delay their graduation dates. Eleven 
percent of students reported withdrawing from classes during the pandemic, and 50% of students 
surveyed reported that they studied less and were less successful academically during the 
pandemic. Students also experienced a reduction in their income during the pandemic, with 31% 
receiving fewer wages and around 40% of students losing a job or internship opportunity. 

Most college students experienced both internal and external stressors that impacted their 
experience with teaching and learning during the COVID-19 pandemic (Bono et al., 2020). 
Stressors are identified as events or phenomena that negatively affect a person's wellbeing (Li et 
al., 2020). During the COVID-19 pandemic, students reported an impact on their mental health 
due to pandemic-related stressors. Students experienced the effects of quarantine isolation, 
illness of self or family members, and financial insecurity (Bono et al., 2020, Aucejo et al., 
2020). 

During the pandemic, students reported stressors related to technology issues that 
included hardware or software problems. Some students did not have equitable access to internet 
connectivity. Some shared a device with another family member or had to borrow a device from 
their university. Some students reported difficulty attending synchronous courses due to 
bandwidth issues caused by multiple users within their households. One-quarter of students 
surveyed by Means et al. (2020) reported hardware and software problems that impacted their 
ability to experience an online course. 

The most pervasive problem reported by students during online learning during COVID- 
19 was the lack of motivation to participate in and complete the course (Means, 2020). Meeter et 
al. (2020) reported that students surveyed attended fewer synchronous class meetings, reported 
less engagement within those meetings, and tracked less time studying than they reported for 
pre-COVID 19 face-to-face courses. In the survey of students reported by Means (2020), 79% of 
students cited motivation as an issue. Fifty-seven percent of respondents indicated that 
maintaining interest in the course content was an issue during COVID-19. 

Students in online courses reported feeling disconnected from their peers and the 
instructors in their online courses, specifically missing the chance to collaborate with peers and 
feeling part of the class community (Means, 2020). Students have expressed the importance of 
peers to support learning and engagement during the face-to-face iterations of the maker courses 
discussed in this paper. Past research on this maker course has discussed the vital importance of 
peer tutoring and other peer input in developing content learning and understanding the course's 
organization (Cohen et al., 2020). Collaborative learning, where peers equal in both knowledge 
and experience work together to complete a task (Damon et al., 1989), is another social aspect in 
the face-to-face maker course that students reported as important. Students formed learning 
groups in previous face-to-face offerings of the course and often worked together in these 
collaborative groups during the in-person lab sessions. Many students reported this sense of 
community and learning within this community as one of the most important takeaways of their 
experience with the course (Cohen et al., 2020). Students in face-to-face iterations of this course 
appreciated the opportunity to see what their classmates were making and using that sharing as 
an opportunity to inform their own choices in which projects they completed. Students 
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eventually pushed one another to continue along a certain path or pursue a new skill (Cohen et 
al., 2019) 


Methods 


This research design is an exploratory case study, with the unit of analysis being the 
whole class. 


Participants 


This course included 13 masters and 9 undergraduate students whose career and 
education goals spanned from K-12 teaching to coaching sports to corporate instructional design. 
Ten master's students and 2 undergraduates consented to participate in the study. Many of the 
students also shared that they were either starting or managing small businesses while attending 
the course. Most class members did not have any experience with maker-centered learning at the 
outset of the course. Though most of the participants had previous experience with online 
learning and expressed comfort with the online meeting format, they were less familiar with the 
mastery-based nature of the course, the unique LMS used to manage course content, and the 
student-centered approach implemented by the course designers. 


Data Collection 


Data were collected during the Fall 2020 semester and primarily consisted of student 
writing and midterm one-on-one check-ins with the professor. Student writing was prompted 
within regular classroom assignments and final exams. Midterm check-ins were conducted by 
the professor via videoconference and were recorded and transcribed for analysis. 


Data Analysis 


Researchers used Dedoose, a web-based software product, to organize and code the 
qualitative data. Inductive line-by-line coding was initially done by two of the research team to 
identify major themes (Saldafia, 2021). The researchers coded sections of the data, engaged in a 
discussion to reach an agreement about codes, and then researchers coded the remaining data. 


Results 
Thematic analysis of the data revealed that COVID-19 related stressors, peer-based 
learning, making remotely, and navigating the unique course design were major themes 


impacting student experience of the maker course online during the COVID-19 pandemic 
closure. 
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COVID-19 Related Stressors 


Participants shared issues related to health and wellness, stressors created by COVID-19 
that impacted their experience of the course. Several students mentioned that they had been ill 
themselves and that their illness—though not serious—had hampered their ability to finish some 
of the module content. Many students mentioned that they had family members or friends who 
were ill with varying degrees of seriousness. Most students shared their experience with health- 
related issues and how they impacted their interaction with the course during the one-on-one 
midterm meeting with the course instructor. Some students shared their experiences in a course 
reflection: 


Unfortunately, I don't feel that much can be done for what I truly need to stay on track. 
It's because it's impossible to control, but I will still share what I need. I need the 
mounting deaths and loss that have occurred in my family to stop or at least slow down. 
You see, since the start of the class, I have lost about 15 family members. This week it 
was my dear cousin's wife, who is a pediatrician. Two weeks ago, it was my dear friend's 
mother. These losses have sometimes caused me to sit and do nothing. It has been 
overwhelming. What helps is having a moment to regroup, but it has caused me to fall 
behind with regard to deadlines. 


While participants did share technology-related stressors that hindered their ability to 
fully participate in synchronous class sessions or manage their coursework, these remarks did not 
often appear in the data. Most students expressed being comfortable using the LMS for the 
course. Most students also expressed being well prepared with software, hardware, and internet 
connectivity for online work after their experience in other online courses throughout their 
college experience. 


Peer-based Learning 


Participants described satisfaction with experiences with decentering that were designed 
as part of the course. Still, many students shared what they perceived as missed opportunities for 
peer collaboration and tutoring that occur naturally in an in-person classroom environment. Past 
iterations of this course in a face-to-face environment have identified the importance of 
collaborative learning and peer tutoring to the students (Cohen et al., 2020). As one student 
expressed, the fully online format "limited the organic connection to my classmates and 
instructors." 

Many students recognized that they would have benefited from peer collaboration during 
the course and missed that possibility during the online version of the course. Several students 
indicated that participating in class was difficult as they felt anxiety whenever they attempted to 
share and were interrupted by or interrupted another student. Some students mentioned that they 
would have liked to hear and see their classmates more and were disappointed that more people 
did not chime or turn on their video during synchronous sessions. Students felt that the 
synchronous online course meetings were not ideal for building collaborative relationships. 
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Several students mentioned that increased use of small breakout groups regularly may have 
increased their connection to their classmates and their comfort level in sharing during 
synchronous classes. One student remarked, 


I would have really benefited from more opportunities to collaborate and interact with my 
peers. I know that is a normal component of the course in the face-to-face setting, but I 
believe I had some really cool classmates and could have benefited from conducting 
more discussions and projects with them. 


Some students suggested tools that they believed would help them communicate and collaborate 
with one another during the online course, typically a forum that "would be very helpful in case 
we get stuck in a module and need to reach out to our peers for help." 


Experiences with Learning about Making in a Remote-Based Environment 


Students overall reported a positive experience interacting with maker-based learning in 
this online version of the course. Several students pointed out that the pandemic had highlighted 
the need for schools to shift education towards a more student-centered approach. Some 
mentioned the need for more educational experiences that encouraged problem-solving and gave 
examples of how working through problems with both the modules and with logistical aspects of 
the course helped them be more engaged and motivated to complete the course. 

Though the skill- and disposition-based benefits of making resonated with the students, 
the practical aspects of remote maker-based learning were more of a challenge. Students 
expressed that access to equipment, materials, and supplies did present a barrier that impacted 
their experience with the course. Many students voiced difficulty finding the time to come to 
campus to pick up kits for home use. Several also indicated travel, health, and wellness-related 
issues that kept them from making the trip to pick up supplies or schedule an appointment to use 
the equipment located in the lab, which was essential to completing many of the modules. 

Nearly every student indicated that what they have learned about making will impact the 
way they think and how they will teach, learn, and engage in many other activities in their lives. 
In their final exam remarks, several students quoted a story told by a classmate that described a 
student who misbehaved in every class except when they were asked to work with others and 
complete hands-on activities. This story was used as an example by several students to show how 
they think making can empower students and help them learn 


Experience with Course Design 

Most students indicated a positive interaction with the course and felt that it was designed 
well for online learning even though they thought it would be better in a face-to-face situation. 
One student wrote: 

This was the best online class I have taken. It was very interactive and felt the most like 


the students mattered to the instructors. The struggle to transition an in-person class to 
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feel natively virtual is difficult no matter how great the content. I am still learning how to 
do this and will be taking many ideas from this class. 


Most students expressed appreciation for the modular design of the course and felt empowered 
by the ability to make choices about which modules they attempted. But many of the students 
who appreciated the autonomy of choosing their own modules strongly indicated that managing 
coursework within this course design required additional time and attention. Some of the 
students expressed difficulty tracking module completion independently. Several students 
detailed the management activities they engaged in to track progress and complete the modules 
on time. Some students made suggestions for strategies that the instructors could provide to help 
students remain on track to make adequate progress in the future. The following was a typical 
suggestion: 


One proposed idea is to have a weekly progress report that would give me guidance on 
the number of remaining modules in accordance with the time remaining for the 
semester. Also, the idea of a one-on-one meeting with the professor is great as it provides 
better feedback and makes room for a conversation to gain insight on the progress and 
potential questions. 


Most students in the online version of the course indicated that time management on a 
personal level impacted their experience with the course. 


Discussions and Implications 


Data made it clear that students' experience of taking the maker-centered learning course 
and their interaction with making were impacted by taking the course online. Stressors that 
accompanied learning during the COVID-19 pandemic further complicated their experience in 
the course. 


Experience with Maker-Centered Learning 


While students reported enjoying the course overall and understood the skills and 
disposition-based benefits of making, they found it difficult to connect to the practical elements 
of making in the online environment. Students in the online version of the course expressed 
frustration with access to equipment and materials to complete the maker activities during the 
course. Students reported challenges to making appointments to visit the lab to pick up kits to 
use at home and also reported challenges to making appointments to do their work in the lab 
space. Students missed the opportunities to work with an instructor on challenging modules. 
Future maker-based learning courses should consider the accessibility of materials, equipment, 
and supplies and explore additional means to make the lab experience more accessible to 
students. 
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Collaborative Learning 


In previous research into maker-based educator preparation courses, peer teaching and 
learning and collaborative learning were important elements that evolved throughout the course 
delivery and were highly valued by the students involved (Cohen et al., 2017; Wohlwend et al., 
2018). Future research may examine ways to embed peer teaching and collaborative learning in 
an online maker-based learning course. 

Many students in the course were disappointed the lack of opportunities for peer teaching 
and learning and collaborative learning that they experienced during the online version of the 
course. As online learning continues within the backdrop of COVID-19 and beyond the 
pandemic, course designers should consider the logistics and potential benefits of embedding 
peer-based learning experiences effectively into an online learning situation for a maker-based 
course. 


COVID-19 Stressors 


Concerns about travel, health safety issues, and scheduling constraints were all indicated 
as stressors that impacted students' perceived access to what they needed in order to complete the 
course. Students reported that personal issues had a real impact on their interaction with peers, 
teachers, and course instruction. These types of concerns were not echoed by students in the past, 
face-to-face, iterations of the course, but appear to be consistent with emerging research on the 
impact of COVID-19 stressors on students (e.g., Fruehwirth et al., 2021). Future research may 
focus on emergency situations like the COVID-19 pandemic response and consider ways that 
academic institutions can prepare students to better manage personal issues and stressors so they 
may reduce their impact on education. 


Course Design 


Data indicate that students were not hampered by the modular design of the course and 
were able to use technology effectively to join the class and to track and complete modules. 
Students reported satisfaction with the modular, mastery-based design of the course, and several 
indicated that the course was the gold standard for online learning in their experience. But some 
students self-reported that they lacked self-regulation skills to maintain progress and stay 
motivated to complete course materials. Students’ difficulties with self-regulation in online 
learning environments is not uncommon (Cho & Shen, 2013), and further exploration of the 
nature of the issues impacting students’ self-regulation—and potential design-based solutions— 
require further investigation. 
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Abstract 


Developing projects is an integral task in course work for students at universities. 
Extensive projects span over semesters and hence are incrementally worked upon by multiple 
teams of students. Ideally, projects are documented and structured to be readily accessible to 
future students who may choose to continue the project, with features that emphasize the local 
community, university, or course structure. The Research and Projects Portal (RAPP) is a 
platform enabling students to post both their completed and ongoing projects with all the 
resources and tools used. Students can access RAPP to understand the contributions of past 
students and may choose to extend the projects that fall within their domain of interest. Industry 
collaborators and faculty from other departments could document their ideas on RAPP for 
students to engage in. Actively curated, the portal enables instructors to label the projects which 
helps students to filter and browse across the portal with ease. RAPP enhances the collaboration 
of students across semesters by supporting their documentation hence enabling longevity of 
projects. 


Introduction 


Students engage in multiple projects during their course work at universities every year. 
Most of these projects are later abandoned because the students ran out of time, or they could not 
accumulate all the resources they needed in a semester span. Inspired by “Communities of 
Practice” (Wenger, 2018; Wenger et al., 2010), this paper presents the idea and application that 
is developed at University of North Texas (UNT) to address this issue. A survey conducted on 
the campus indicated that projects done by students every semester are available on individual git 
repositories. New students, who are in the process of picking a project for the semester needed a 
common platform where all the past student projects of the department are archived along with 
their documentation. This lack of aggregation of resources hindered students to refer past work, 
track resources and choose to increment an existing project. 


Students working on multiple projects had to navigate through multiple platforms to 
maintain documentation and track the status of each project. Recruiting, to onboard new students 
for a project was very elaborate where the concerned instructors and students sent out email 
communication along with putting up flyers on department dashboards. On the other hand, new 
students in search of open projects needed a community to tap into. Similar kind of 
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communication was taking place with students needing funds for their projects or sponsors 
seeking good projects. Additionally, an interface that could bridge the gap between industry 
collaborators and the department student projects was not available. Industry collaborators, given 
a better communication platform could bring in value-added solutions, resources and provide a 
perspective on the current market necessities in any given domain (Silva et al., 2018; Fernandes 
et al., 2018). This survey led us to create a better design which addresses the challenges by 
having an archival of past projects, support organized documentation with all the relevant 
resources and tools, curate the project list for end users (refer Figure 1). 


fella cella) 
archived A 
projects for 
reference 
: ad Le 
(wc) | [| 
Seek students , (oo) 
to work for '=s —<- 
projects Ps (mc) —* 
— 
= 
G- 
Figure 1. RAPP Features Figure 2. RAPP Architecture 


The emphasis on having long-term student projects that span over multiple semesters was 
experimented at Grinnell college in Iowa. They have redesigned their ‘Software Design’ course 
to enable students to engage in multi-semester, community service projects (Davis & Rebelsky, 
2019). These changes were done to accustom students to not just the development of real-time 
applications that served the local non-profits but also to introduce them to the terrain of software 
project maintenance. The student teams were using Git to maintain code versioning along with 
documentation. Alumni mentors were assigned to the student groups who help them navigate 
through design problems and advice on the optimal technical solutions. This line of work 
displayed promising results in increasing the technical and soft skills of the students (Davis & 
Rebelsky, 2019). 


In recent years, significant work has been done on archiving, evaluating, and assessing 
student projects through various web portals (Li Li et al., 2007; Di Blas et al., 2014). At the 
University of Bradford, a knowledge portal was implemented to support the collaboration of 
MSc student projects (Munive-Hernandez, 2011). This portal supports monitoring of project 
tasks and tracking the milestones achieved. It also enables communication and feedback from 
supervisors. The portal is actively being used for project definition, project planning, literature 
review report, supervisor’s feedback, and development of knowledge repository. Similar efforts 
were initiated in 2003, at the University of Pretoria in South Africa. The web portal was a 
massive success and was also made the official architecture at the university (Pienaar, 2003). 
More recently in 2018, Monash University in Australia developed and launched an online 
research portal that grew to be accessed by almost ten thousand people and implemented 
throughout the campus (Rodafinos et al., 2018). Another example of the successful 
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implementation of an academic research portal occurred at Vanderbilt University. Features of 
this portal included research support, and guidance for future projects (Harris et al., 2011). The 
success of similar efforts provides a reference for the future success of RAPP. 


Figure 3. Application Workflow Figure 4. Project Statuses 


While many web portals lag in displaying incomplete projects, the goal of RAPP is to 
build a website that is actively curated with a list of student research projects that are active and 
well suited for students and faculty to continue. An archive is also intended to showcase the 
research and class projects that students work on to excite prospective students, interested 
faculty, and prospective donors (Rodafinos et al., 2018). The user-friendly graphical user 
interface of RAPP enables professors and students to spread awareness of what they are currently 
working on and tag their projects as “actively recruiting” to seek students. While at the same 
time, students can browse through to see the projects that belong to their specific domain of 
interest and find the materials to work with or a faculty member best suited to advise them. 


While still in the formulation phase, an idea can be proposed on the portal to become a 
prospective project. This feature of RAPP makes it possible for students to collaborate with a 
professor and arrive at a project plan that best suits both parties. Furthermore, this invites 
industry collaborators to bring in their value-added suggestions to these proposals or proactively 
fund a project. The influence of university-industry collaboration on innovation has gained a lot 
of attention in recent times due to its positive impact (Barnes et al., 2000; Hansen et al., 2017; 
Kukreja et al., 2020; Szabo, 2014; Gorlatova et al., 2013). The portal’s user community includes 
students, faculty, and industry collaborators. 


All the users in the community should sign up to the portal to gain access to all the 
features provided. To add a new project, one should upload a brief description, source code link, 
documents, and output snippets of their projects. The portal then prompts the user to tag key 
names related to the project which are later used in filtering the projects list. Signed up users can 
browse through the list of projects and filter based on tags assigned to the projects. Students are 
benefitted by the portal, by being able to upload their ideas/projects and seek out projects that 
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fall within their domain of interest. Faculty and industry collaborators are empowered by the 
portal to be able to reach out to students actively seeking projects and resources, by hosting 
projects that are to be worked upon. 


Implementation and Methods 


RAPP is developed using the MERN stack, designed to run on Node.js and Express.js for 
server-side operations (refer Figure 2 & Figure 3). The front end of this portal uses a react and 
redux combination. Redux is a predictable state container that handles server and client 
interactions. The client relies on the index.js file and app.js files for redux setup and client-side 
routing. The MongoDB is responsible for storing and serving all the user and project data. User 
accounts and project listings are stored on different tables within the database. 

This portal leverages the EUID-based login (LDAP) system implemented by UNT, based 
on Microsoft Active Directory service. Using this unified login information service, any 
authorized user with valid UNT credentials will be able to login to the portal. Administrator role 
is assigned to users who are responsible for the maintenance and health check of the portal. An 
admin user is also accountable for reviewing the proposed projects. RAPP has a responsive web 
design enabling it to be rendered and fit into any screen resolution, size, and orientation of the 
user’s device. HTML5, CSS, JavaScript and Bootstrap v4.0 are used to enhance the frontend 
experience of the user. Bootstrap is the most popular CSS framework for developing responsive, 
mobile-first projects on the web. 


Results 


RAPP enables the user to navigate through multiple tabs that facilitate and host the 
features. This web portal allows any member of UNT to submit projects or ideas for review by a 
site administrator. Once an administrator approves a submission, it becomes visible among the 
list of all projects. This list has a robust set of search and filter functions, allowing members to 
view and share a completely customizable set of projects. Contact information is included with 
every project, so adopting an existing project is easy. Projects can be edited by their owners, and 
by site administrators, so changes in status and team ownership are easily updated. Administrator 
has a suite of tools to curate and manage the database of projects. These tools include approving 
or removing newly submitted projects, batch editing multiple projects, exporting the current 
database as a spreadsheet, and importing multiple projects at once from a spreadsheet. 
Administrators also can promote and demote other site users to administrator status. 

Functional features available on RAPP are described below. 

a) Add Project 

A user starts with uploading his idea to the portal via this form. User is expected to 

feed in the information of the project (refer Figure 5.1 & Figure 5.2). Actively recruiting label, 
is a check box which lets the users understand if there are open positions in the project. 
GitHub link label is a text box to feed in the link to the source repository. Cover Image label is 
an upload option for a .jpg or .png file that serves as a cover image for this project on the 
portal. 
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Submit a New Project or Idea 


ew submis eviewed by an administrator before becoming visible on this site 


Figure 5.1 Add Project Form Part1 


= 
Figure 5.2 Add Project Form Part2 


b) Edit project: 

Users are provided with an access to update the projects owned by them. This serves as 
a crucial feature for the portal since these user activities help actively curate the project list. This 
form is same as the “Add project” form since the portal provides complete access to the user to 
update all sections of the project. 

c) Dashboard for browsing projects: 

A dashboard with all the projects listed except for the archived projects. The page 
comes with user-friendly interface for the user to be able to navigate through with ease (refer 
Figure 6.1 & Figure 6.2 & Figure 6.3). A search bar is available for the user to feed in the project 
name. This helps the application to find the desired user project from all the hosted projects. A 
list of project status filters (refer Figure 4) is provided to the user to choose from. The filters are 
segregated based on three categories namely, development status, administrator approval status 
and recruiting status. 
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Machine Learning to Identify Cats April 11, 2021 


What cat is that? Robots have the answer! 


Pre 
A a 
Pa Proposal #cat machine rs 
Stoppe a 
ed 
Algorithm to Detect Animal Type April 11, 2021 
a Hanna Flore 
C f h det hat kind of animal made that 
f a 
Not Recruiting Qo Proposal Recruiting #animals #dog #machinelearning cat 
Tag Filters 
3. Stock Market Indicator April 8, 2021 
asmine i 
fom Using ten years’ worth of daily stock price data along with the resulting 
technical indicators, we utilized the first 7.5 years as training data in our 
Figure 6.1 Dashboard to Browse Projects 
Machine Learning to Identify Cats April 11, 2021 
Hanna Flores 
What cat is that? Robots have the answer! \ University of North Texas 
More Details : 
UNT Research and Project Portal 


Proposal #cats #machine learning 


Figure 6.2 Project Tile Figure 6.3 Application Menu Bar 


The project development status reflects the implementation status of a project. The 
administrator approval status depicts whether the project had been approved by the admin of the 
portal or not. This filter is used by the user to check if the projects onboarded by him/her is 
approved by the admin. The user can leverage the filters in the project recruiting status section to 
filter or filter out the projects that are actively recruiting. All the projects, except for archived 
status that fit the search and filter criteria are listed. Each project in the list is contained in a tile 
which holds all the details of the project. The principal feature of RAPP is to curate the user 
content. To be more specific, projects that are recently updated, active or actively seeking 
resources will be prioritized and will be on the top of the list. 
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Tag Filters Reset 


Figure 7. 1 Include Tag Filters 


Tag Filters Reset 


+ 


Figure 8.1 Exclude Tag Filters 


Algorithm to Detect Animal Type April 11, 2021 


Proposal Recruiting #animals dog #machine lear 


Figure 7. 2 Include Tag Results 


No Projects 


Figure 8. 2 Exclude Tag Results 


Each project tile contains a “More Details” link that when clicked upon would help the 
user navigate to a different page where more elaborate details of the project are available. 
Thumbnail picture attached to the project is projected on the project tile for a visual reference. 
Filters to search for projects by their tags. While creating a project, the user is entitled to assign 
tags to the project. These tags reflect the technology or domain specific keywords related to the 
project. Furthermore, the results can be narrowed by specifying tags you do not want to include 
(R\refer Figure 7.1 & Figure 7.2) and exclude (refer Figure 8.1 & Figure 8.2) in your search. 


Login 


Figure 9.1 Login Page 


d) Maintain user profile: 


Hey there, Noelle. 


Personal Information 


Name: Noelle Mansilla 


Email: NoelleMansilla@my.unt.edu 


Your Projects 


example project 1 


example project 2 


oo 


Figure 9.2 User Profile 


The portal provides a profile page for the logged in user to display all the projects under 
the user’s ownership (refer Figure 9.1 & Figure 9.2). This helps the user to track the statuses of 
all his projects and have easy access to navigate to those specific projects and update the content 
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and details. The absence of this feature forces the user to navigate to the dashboard and search 
for these projects. From this profile page, the user can navigate (view), edit, and delete the 
project. 

e) Admin Dashboard: 

The admin privileges enable a user to have access to a set of features that help manage 
the portal. Admin features include import and export of projects; privilege to approve, deny and 
edit projects; view all the users and have the privilege to assign or revoke admin role to any user 
on the system. The Pending projects tab helps user navigate to the admin dashboard where a list 
of projects, that are to be reviewed by the admin are available (refer Figure 10 & Figure 11). 
Each project tile in this dashboard comes with an approve button and a reject button. Upon 
approval, the project will be made available to all the users on the portal. In case of rejection, the 
project is not available on the projects dashboard for the users to view and work on it. 


Edit Projects 


Admin Pending Projects 
Navigation 


Identify Slugs From Images 


use Al to develop the best cookie recipe 


Figure 10. Admin Dashboard Figure 11. Projects Edit Page for Admin 


When there are multiple projects that are to be onboarded to the portal, it is a tenuous 
process for the user to manually type in all the details and upload the project one by one. In such 
scenarios, the user may reach out to the admin and provide all the project details in an import 
template. The template is available for download on the import page. The admin is supposed to 
upload all the projects through the template alone for this bulk project upload to be successful. 
When changes are to be applied to multiple projects, it is ineffective for the admin to navigate to 
individual projects and update the changes. In such instances, admin has the flexibility to export 
all the projects and update the content in excel at ease. Post applying the changes, admin can 
import the same for the portal to reflect the updated content. The export also helps to take a 
backup easily, in case of system migration or upgrade. 
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The admin has a privilege to manipulate the tags on a project along with its status. An 
option to delete the project is also provided here (refer Figure 13). This gives the admin complete 
access over the projects, so that the portal does not encourage malicious content, projects on the 
portals hold relevant tags that enable effective filtering, and the status of each project is 
reflecting its needs or implementation status as expected. The admin user will be able to view all 
the users registered on the portal. This feature enables the admin to provide or revoke admin 
access to other users on the portal (refer Figure 12). 


All Users 


Editing Selected Projects 


Delete Projects 


Update Projects 


Figure 12. View Users Figure 13. Admin projects edit page 


Evaluation 


The portal is launched in April 2021 with a primary goal to be available for the “bio- 
medical AI lab” summer program at UNT. The “UNT AI Summer Research Program” brings 
together students from a variety of Al-related academic programs to supplement their traditional 
course-based educational experiences with focused, project-oriented research efforts. Faculty and 
students with promising ideas uploaded their proposals on the portal. The members of the review 
panel were given admin roles on the portal to review the proposal submissions and approve them 
so that they could become prospective projects. The rejected proposals are not available for the 
users on the “Projects” dashboard. We had a team of students to administer a survey on the user 
experience of RAPP. This survey involved extensive feedback collection in weekly lab meetings 
throughout the summer program. The prime objective of this survey is to understand if the portal 
provides ease to the user while navigating through different tabs, if the forms provided were 
wholesome and relevant, if there were any glitches throughout the experience, if the filters and 
tags provide relevant curated content that helps the end user, if the aesthetics of the portal were 
appealing. The portal received an extremely positive reception. 

Admin users were able to manipulate the tags on any given project with ease along with 
being able to edit the approval status, recruiting status and project status. This meant that they 
could refine the content with ease so that search results for the end users would be appropriately 
curated. Besides, the admins could monitor for any rogue tags and censor them to maintain the 
integrity of the portal. The import and export project options also received critical acclaim since 
the admin users were able to instantly onboard multiple projects effortlessly from past summer 
programs. With more proposal submissions, the admins were able to appoint new admins and 
scaleup the panel and remove their admin access when the workload receded. This was made 
possible with the ‘Make Admin’ and ‘Remove Admin’ options that are available on the “All 
Users” page accessible to all admins. Furthermore, the portal provided the admin with a 
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prioritized queue of pending approvals that would make sure that the users who submitted their 
projects early would not have to wait for a longer period to receive the approval status. The 
survey proved that a feature as subtle as this could bring in a magnificent shift in user 
experience. Prior to this, the review panel had to track their mails and refer the tracker updated 
by the students and found it extremely tedious in identifying the order of approval. At times, this 
delay would lead to vexation of many proposers. 


The faculty in the lab got back to the survey team that they were able to easily recruit 
students for open projects since it was as easy as marking the project as “Actively Recruiting”. 
Students were able to effortlessly filter out projects that had open positions through the portal 
and then reach out to the team with the contact information available on the project page. The 
well-organized portal facilitates grant writings for faculty since they could access all the 
resources in a centralized location. The fact that the portal does not need a separate registration 
process was an open invitation to people from all walks of academia at UNT, to propose ideas 
and manage their projects. This feature has created a substantial scope for the expansion of the 
portal to wider audience soon at UNT. 


Discussion 


Unlike GitHub, RAPP’s goal is to go beyond just hosting projects. RAPP allows faculty 
and students to connect over unfinished projects, while still being able to display completed 
projects. To achieve this goal, we used appropriate client-side design tools to make the collected 
data user-friendly for browsing. Current project features include adding, searching, viewing, and 
filtering projects. They can also add tags, images, GitHub links, contact information, and upload 
files to projects. The fact that RAPP provides actively curated content along with dynamic tags, 
makes it a robust application that brings in a difference to the user experience. Taking it further, 
RAPP could provide a feature for the user to compose and send out an email to the project team 
directly. A chatbot that interacts with the user and suggests filters or projects would enhance the 
user experience further. A feature that would enable all the logged in users to participate in a 
conversation via text messages could push the collaboration to the next level. Though RAPP is 
launched as a pilot for the summer research programs at UNT, the goal is to refine and expand 
this application to be used widely throughout the campus. 


Conclusion 


Student projects with a substantial scope need incremental development, that spans over 
multiple semesters. Lack of a well-built model to bridge these efforts could obstruct the 
continuation of the projects. RAPP addresses this issue by enabling students to host projects with 
documentation, and respective resources. While completed and paused projects help with 
retrospection, new and active projects could be tagged with appropriate labels that results in 
curation of content hence encouraging users to find most active projects or projects with open 
positions. These features help the university’s faculty and students to find resources and related 
work while at the same time they encourage industry collaborators to bring in contributions. 
RAPP has a profound impact on its user community by creating a sustainable and collaborative 
environment for student work to thrive. 
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Abstract 


Continuing Education course development can be a complex environment requiring rapid 
response to change and development of multiple projects at once. This paper describes the 
development of an instructional design model based on a series of decisions made during the 
design of an array of six pregnancy and postpartum fitness courses for different sets of learners 
with specific identities and learning goals and objectives. The collected data of the projects from 
task logs, project management documents, communications, and project reflections of the natural 
work setting were interpreted and generalized to make causal inferences. In this context, 
behavior patterns of the instructional designers emerged. This design case presents the action- 
oriented process of inquiry employing Soft Systems Methodology (SSM) to structure discussion 
about the situation and identify the repetitive procedural decisions to develop an instructional 
design model for higher education that could systematically guide instructional designers 
through rapid iterative development that employs user stories and Weick’s sensemaking. The 
precedent is revealed in the User Story Approximation Model (USAM), which when replicated 
will result in newly designed outcomes in drastically reduced production time. 
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Introduction 


Seeking knowledge to develop new competencies is no longer a sporadic activity in our 
professional lives. Shifting trends in skill acquisition to increase employability have created a 
demand for high quality continuing education courses and programs at colleges and universities. 
The instructional designers working at small institutions often have the unfortunate task of 
serving two masters: degree programs and continuing education programs. Further, course 
development can be especially chaotic in the continuing education environment where the 
potential for development is endless, and where any number of stakeholders and their requests 
can create challenges in managing priorities. The result is time constraints on the design and 
development process that ask instructional design units to find ways to shorten the time of course 
development. For continuing education programs to be successful, learning design must be 
structured to allow for flexibility and entrepreneurial thinking. Because of the need for design 
methodologies which are more efficient, while maintaining or enhancing effectiveness, rapid 
prototyping (RP) was evaluated as an instructional design strategy in the 1990’s and numerous 
studies since that time have indicated that rapid prototyping improves instructional project 
quality and customer satisfaction while reducing production time. 


In the discussion of RP, the use of reusable learning objects (RLOs) in course design 
usually reflects a conversation about reusable content. Learning objects that are made agnostic 
in such a way as to make the learning object viable in more than one course or program 
involving an interdisciplinary thought. The idea behind this approach is that the viability of the 
learning object is extended by the design decisions made in the development of the object which 
reduced the time to production. These decisions when formalized can be replicated to create 
RLOs. This approach to reusability focuses on “filling the closet” with a myriad of choices to use 
in each newly designed course. Unfortunately, this approach “fills the closet” or RLO repository 
with objects that need to be ordered, organized, and maintained, which may not produce the 
reduced production times some instructional design units require. 


With the acceptance of our need to work faster, we sought a development model and a 
technique that would increase our production. Torrence’s Agile for Instructional Designers 
(2019) borrows heavily from the AGILE approach developed by Sutherland and Schwaber, 
which relies on a philosophy, or “Manifesto” as Sutherland and Schwaber call it, and which 
Torrence modifies for use in instructional design. This manifesto guides all decision making. 
AGILE instructional design represents a form of development where design teams apply project 
management methodologies originating from software development to the practices of 
instructional design to rapidly develop courses. Emphasized in these practices is an iterative 
process which provides the opportunity to use collaboration, feedback, and iterations (versions) 
to reduce time while producing a more successful and valued product. 


Course development is framed from the perspective of the learner. The course plan 
addresses the needs of the learner and how they will engage and interact with the course. The 
creation of the course is divided into modular elements or chunks. The individual modules are 
designed, and these portions are reviewed early on by the collaborators to produce a higher 
quality course more quickly while reducing the need for last minute revisions. 


Most instructional design models devise the design of learning objects as new designs 
from the “blank page”. We propose the implementation of a rapid prototyping approach through 
the construction of course modules from reusable component templates. This concept of 
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reusability was first introduced in instructional design processes with an object-orientation where 
a system of best practices and design principles guide the development of reusable objects 
(Boyle, 2009). We proposed a further modification of the design principles based on Atomic 
Design (Frost, 2016) that allows atomic construction of design templates for use in course 
development which creates course prototypes within days of initial meeting with SMEs rather 
than weeks and full production to launch to occur within weeks. 


Once a course has been constructed, an iterative cycle may begin to create successive 
versions of the course for other programs and other users. This process begins with a 
reconsideration of the “user story” and provides an opportunity to employ Weick’s sensemaking 
(Mills, et al, 2010), which “is about understanding how different meanings are assigned to the 
same event,” to revise the course. Weick’s sensemaking guides the revision and how collective 
decisions are made at each stage of sensemaking to re-envision the course for a new user story— 
a new set of learners. Through these critical design decisions an array of courses that address 
different sets of learners may be rapidly constructed. 


The purpose of this design case was to describe the development of an instructional 
design model to guide designers systematically and effectively throughout the design and 
implementation process of modular design with reusable design objects (RDOs) employing 
AGILE decision-making principles and Weick’s sensemaking to rapidly produce courses. This 
user story design model is intended to result in an array of rapidly created continuing education 
courses with an appropriate blend of individualized content and learning activities that 
coherently provide meaningful learning experiences for multiple user stories and contexts. 


“Models make possible a systemic approach to design in that they intentionally 
lead designers to balance considerations of varied critical factors. Models 
incorporate both theoretical and empirical research in related fields. Further, the 
iterative process of evaluation and modification leads to improvement of practice, 
pointing to the potential design to stimulate continued progress in a larger 
context” (Lee, et al, 2017). 


The design model was developed interrogatively. It questions who, why, and how 
particular reusable design objects (RDOs) should be designed, selected, or modified in a 
learning arrangement. Until now, no prior instructional design models have investigated 
the identification of user stories to develop multiple variations or versions of courses in 
succession through modular design and sensemaking. The process for constructing a user 
story design model reported in this design case, and its implications for improvement of 
continuing education course-level development and instructional design practice are 
considered to contribute to the growing literature of learning design. 


Conceptual Modelling 


Conceptual modelling may be defined in the simplest terms as the process of developing 
a graphical representation that provides an understanding of collaborative problem solving 
within a system for the different stakeholders involved. Conceptual modelling requires decision 
making to identify the scope and level of detail, aspects to include and exclude, and the 
objectives, inputs, outputs, content, assumptions, and appropriate simplifications to be contained 
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within the model. A well-designed conceptual model establishes a common language among 
stakeholders that facilitates planning, design, and evaluation. 


What is a conceptual model? 


According to Powell-Morse (2017), “a conceptual model is a representation of a system 
that uses concepts and ideas to form said representation, [and] is used as a way to describe the 
physical or social aspects of the world in an abstract way.” Since a conceptual model may 
incorporate representation of both behavior and data at the same time, it should fulfill four 
fundamental objectives in its construction: 


e Enhancement of understanding of the representative system, 

e Promotion of efficient conveyance of system details between stakeholders, including the 
key entities of the system (person, place, concept, event, and relationships), 

e Provision of a point of reference for designers to gather concepts and sub-concepts, and 

e Documentation of the system for future reference. 


Conceptual models may serve to regulate behavior within a process and establish entities 
and concepts that help eliminate unsuspected outcomes. They define scope which aids in time 
management and scheduling, and serve as high-level understanding for managers and executives 
who may not be familiar with the minutiae of the design process (Powell-Morse, 2017). 


Development of a Conceptual Model 


Though there are many approaches to the development of conceptual models, Soft 
Systems Methodology (SSM) provides a systems thinking approach to operationalizing 
conceptual modelling. SSM is an action-oriented process of inquiry for addressing multiple 
views of reality in a situation to derive purpose from actions to inform processes (Checkland & 
Poulter, 2020). Kotiadis and Robinson (2008) derived a three-stage process and accompanying 
sub-processes for the creation of conceptual models from SSM. When employed, these processes 
create the interactions necessary for conceptual model construction. The stages included in the 
processes are knowledge acquisition, model abstraction, and arbitration. Knowledge acquisition 
(KA) may be divided into two parts and their resulting sub-processes. KA begins with 
constructing a rich picture that is a holistic representation of the situation under scrutiny. Key 
elements include descriptions of the processes, procedures, and stakeholders. 


The second part of KA may include three analyses: role analysis, social system analysis 
and political system analysis. These analyses provide information about decisions made and 
what part behavior played in the success of the process described in the model. Role analysis 
requires consideration and exploration of “the role of the client (who has caused the study to take 
place), the role of the ‘would be problem solver’ (who wants to do something about the situation) 
and the role of the problem owner [who would like the process to be documented for future use]. 
All or some of these roles may overlap” (Kotiadis & Robinson, 2008). The goal is to answer the 
key question--why is this model necessary to the stakeholders. Once roles are determined, social 
system analysis should take place. Social system analysis considers “the changing interactions of 
roles, norms, and values”. The goal is to answer what affect, actions, and outcomes has each role 
provided. 


Finally, political system analysis enables an understanding of “how power is expressed in 
a particular problematic situation.” Who made key decisions and from where did decision 
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making derive? Where were the points of review, evaluation, and approval within the mapped 
process? “Understanding the roles within a problem situation, typical behavior of the 
stakeholders and the allocation of power can mean that the modeler can manage the stakeholders 
during the conceptual modelling process and arrive at a conceptual model that is agreeable to all, 
desirable and feasible” (Kotiadis & Robinson, 2008). 


Once KA is complete, abstraction, the process of simplification to identify purposeful 
actions, may be made from the collected data. When a resulting origination of workflow of 
purposeful actions is identified, an initial iteration of the conceptual model is created and serves 
as a partial representation of the system description. Simplification is achieved by reducing the 
level of detail to behaviors that may be categorized according to scope. The difficulty in 
abstraction remains developing a balance to provide an accurate portrayal of the situation and 
model objectives. 


The final stage in conceptual modelling is arbitration. The subject matter experts debate 
the situation, using the draft models to provide changes which may improve the process and are 
desirable and feasible, and accommodations between conflicting interests are made, which will 
enable improvement in the process. Once a consensus is reached, the modeler may draw the final 
rendering of the model to be conceptualized. 


Results 


Research on instructional design models may be classified into three different types: 
model development, model validation, and model use. This design case was concerned with 
model development and followed the process methodology designed by Kotiadis and Robinson 
(2008) from SSM and utilizing criteria to establish validity defined by Richey & Klein (2014) 
who noted that ID models may be developed from practical means, utilizing real-life design 
projects. The user story design model in this design case was developed from real-life design 
project data. 


Participants 


The conceptual modelling involved participants composed of the members of the ID team 
(one subject matter expert, two instructional designers, and one administrator). The SME was a 
graduate student and fitness professional with 15 years’ experience in the fitness industry. The 
first instructional designer had 10 years’ experience as an instructional designer and 5 years’ 
experience as an instructor in higher education. The second instructional designer had 20-years’ 
teaching and curricular design experience and 10 years’ experience as an instructional designer. 
The administrator previously completed instructional design work as a 20-year faculty member. 


Knowledge Acquisition, Part 1: Holistic Representation 


The collected design projects of an array of six pregnancy and postpartum fitness courses 
were profiled in scope, resources, and roles. Records of key project data including work logs, 
project management documents, and in progress communications were collected. Extracted from 
the resultant data were timelines of tasks completed, decisions made, people consulted, resources 
used and the extent to which instructional design and institutional standards were addressed and 
met. Problems in the development cycle such as confusion over processes, directions, and 
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language were described including constraints, conflicting information, failures, and 
miscommunications. 


Knowledge Acquisition, Part 2: Analyses of Culture and Disposition of Power 


Roles were explicitly defined from the perspective of expectations, job descriptions, 
prescribed actions, and actual performance. The key roles were identified: role of the client (the 
SME), the role of the problem solver (the ID), and the role of the problem owner (the 
administration). Reactions and opinions from all roles including the amount of time spent in 
problem analysis, how problems and design decisions were addressed through the development 
process, and the combination of decisions and interventions to solve performance problems were 
mapped in the workflow. 


Social system analysis identified interactions between roles and the extent to which roles, 
behavioral norms, and values shifted. The effect that the disposition of power through the 
implementation of AGILE communication processes was defined and how these processes 
affected communication and decision making in development were explored. The influences of 
organizational climate and the impact of the role supervisors had in the development process 
were also explored. Cyclical behaviors were typified including the design, development, and 
review of iterations and versions, and the design behaviors were described through sensemaking 
of successive iterations, and the language of all interactions was clarified. 


Noting that “data triangulation is especially critical in design and development studies 
based upon participants’ recollections of already completed projects” (Jones & Richey, 2014), a 
design case paper describing the workflow was written as a reflective exercise to understand the 
implications of the findings and situate the learning for conceptualization. The design case 
served as a confirming process for the researchers to validate the work completed during 
knowledge acquisition. 


Abstraction 


The scope and level of detail identified in the system description provided by the subject 
matter experts was simplified to task categories often found in instructional design models. A 
textual outline of processes was created. To consider graphical representation of tasks, a set of 
questions was developed to guide the mapping of specific design details as presented in Table 1. 


Table 1 


Interrogative Framework to guide mapping of purposeful actions 


What should students learn about this topic? 

Who are the students for this information (career orientation)? 
What are the desired results for each user story? 

How will the content be developed? 

How will learning objects be modified for each user story? 
How will each user’s learning be accessed? 

What is the order of course development? 

How does rapid prototyping occur? 


PON NAMA WND 
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The base design was created by mapping the specific design details on the Successive 
Approximation Model (Figure 1), which served as a base model for graphic design and a starting 
point for arbitration. The graphical representation of the base model is shown in Figure 2. 


PROTOTYPE 
BACKGROUND 
DESIGN REVIEW 
> oe 


Information SAVVY Project Additional 
a ED 


Preparation Iterative Design Iterative Development 
Phase Phase Phase 


Figure 1. Successive approximation model version 2 (SAM2) process diagram. Adapted from 
Leaving ADDIE for SAM: An Agile Model for Developing the Best Learning Experience (p.40), 
by M. Allen, 2014, Alexandria, VA: American Society for Training & Development. ©2014 by 
the American society for Training & Development. 
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Figure 2. Base model draft to be used for Arbitration. 
Arbitration 


The ID team conducted a retrospective meeting to evaluate the base model draft in 
comparison to the design case. The instructional designers debated the situation as compared to 
the draft model to provide changes which would more accurately reflect the process as it was 


completed. The discussion included recognition that not all process elements are cyclical and not 
all process elements are linear, but a combination of both. A consensus was reached, and a final 
rendering of the model was made, as seen in Figure 3. 
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Figure 3. Final rendering of the User Story Approximation Model (USAM). 


Discussion 


Design and development processes are particularly difficult to navigate and manage, 
especially in environments coping with scarcity of resources and other limitations. These 
processes may be more effectively understood, improved and supported through the development 
of conceptual models. As this model demonstrates, design and development processes involve 
significant amounts of novelty, complexity, and iterations. The model is described in Figure 4. 


The assumptions of the model 

e = The users of this model are an instructional design team (SME, instructional designers, and an 
administrator) for continuing education courses. 

e Courses are designed in 5 modular units with rapid prototyping techniques through AGILE instructional 
design processes. 

e Inthe context of the model, “Goals” refer to what students need to achieve in their real-world context 
after completing the course. “Objectives” refer to what skills and knowledge students will acquire after 
completing the course. 
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Description of the Model 


2.1 Course design 
Instructional Designer analyzes content and 
creates course outline by distributing topics 
into modules. The overall structure of the 


course is determined. Time plan, instructional 2.3 


methods, multimedia assets, feasible learning 
activities, and formative and summative 
assessments are planned. 

2.2 Prototype Development 
Instructional Designer develops prototype, 
including syllabus, course layout and modular 
design, reflecting ID team and institutional 


standards, and the contractual obligations. 2.4 


Instructional designer decides on specific 
learning content to be delivered and through 
what means. Various materials including 
lectures, presentations, images, tables, and 
figures are developed. Instructional designer 
works with SME to produce video assets and 


1. Preparation Phase 
1.1 Initiator delivers a Body of Knowledge. 1.2 Product Owner provides relevant information to 
The SME provides course idea and core construct user story, which includes contractual 
knowledge and works with instructional obligations set by partner institution and their 
designer to develop course name and cultural context. 
description. 1.3 Goals & Objectives 
Instructional designer constructs overall goals for 
the course, which should address what skills 
students should exhibit in career performance. 
2. Initial Course Design Phase 


Instructional Designer and SME work to develop 
formative and summative assessments that relate to 
career orientation of the course, and they are 
placed within the prototype. 

Iterative Review 

Prototype is reviewed by stakeholders and after 
discussing appropriateness with ID team, revision 
occurs. Process is repeated until the ID team and 
institutional standards have been met, contractual 
obligations have been met, and the SME has 
verified all course content is appropriate and 
accurate. 

Alpha Course is completed. 

Course is delivered to stakeholders. Upon 
successful implementation, course review survey 
data is reviewed by ID team for future 
improvements and/or course developments. 


prepares them for embedding in the course. 


phase) 4.3 


4.1 Alpha Course is duplicated. 
At this phase, the Alpha course prototype is 
reviewed in terms of achieving new learning 
goals and objectives. 

4.2 Weick’s sensemaking is applied. 
Instructional designer and SME check to 
confirm if course materials will cover new 


learning objective, if there are new objectives 4.4 


to consider, items to remove, new items to 
develop, and items to modify. 


3. Iterative Preparation Phase (repeated phase) 
3.1 Seek new product owners. 3.3 Goals & Objectives 
The administrator reviews the course and Instructional designer reviews goals and objectives 
attempts to identify other partners for course from Alpha course and revises overall goals for the 
implementation. iterative course, which should address what skills 
3.2 Product owners provides relevant information students should exhibit in career performance. 
to construct user story, which includes 
contractual obligations set by partner 
institution and their cultural context. 
4. Course Iterative Development Phase (repeated 


Iterative Review 

Prototype is reviewed by stakeholders and after 
discussing appropriateness with ID team, revision 
occurs. Process is repeated until the ID team and 
institutional standards have been met, contractual 
obligations have been met, and the SME has 
verified all course content is appropriate and 
accurate. 

Successive Courses are completed. 

Course is delivered to stakeholders. Upon 
successful implementation, course review survey 
data is reviewed by ID team for future 
improvements and/or course developments. 


Figure 4. Description of the Final User Story Approximation Model 
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Conclusion 


Conceptual models define scope and aid in time management in the design and 
development process. Conceptual Modelling activities utilizing the actual task logs kept by 
designers, developers, and even clients and SMEs are invaluable as they provide a record of the 
purposeful actions and decisions made and allow modelers to conceptualize a workflow more 
reflective of documented practice. The final USAM created in this design case evolved 
gradually from the reflective practices associated with AGILE project management 
communications. The target users, scope, and design context were revealed in the model 
assumptions and description. 


Model development is an iterative process. This model may be limited in that it was 
developed from a single case at a small regionally accredited institution with one instructional 
design unit serving degree and continuing education course development needs. Also, since the 
design case specifically focused on the development of an array of continuing education courses 
at a special focus university, the model may include some features that are specific to the career 
context of sport and working with regional, national, and international partners, stakeholders, and 
SMEs. 


Further, every design and development process involves a degree of uncertainty. 
Validation of this model may reveal new activities typically discovered during implementation. 
New complexities may be revealed and should be considered, which will generate and 
communicate new conceptual insights into the design and development process, which may 
contribute to a better depiction of best practices and contribute to new knowledge in the growing 
body of conceptual models for instructional design. 
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Abstract 


In learning design, it has been long understood that a body of knowledge (BOK) is the 
complete set of concepts, terms and activities that make up a professional domain. The literature 
describes the purpose of bodies of knowledge in their role as a requirement for professional 
identity, the development and furtherment of professionalism, and the acclamation and 
dissemination of knowledge to embody shared values and practices. Professional status requires 
the field to monopolize a discrete BOK, but how this knowledge is organized varies from 
profession to profession which may present difficulties in the development of conceptual 
frameworks for continuing education course development where courses are frequently created 
and aligned to developing trends and issues in the field. In this paper, a matrix of professional 
knowledge aggregates in the form of subject matter experts is presented and their expertise is 
considered from the perspective of Anderson and Krathwohl’s (2001) Knowledge Dimensions. 
Five (5) approaches for the development of continuing education programs based on the BOK 
presented by the subject matter expert at the beginning of course development will be shared. 
The benefit of this design approach is to create continuing education programs more successfully 
aligned with professional practices. 
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Introduction 


In learning design, it has been long understood that a body of knowledge (BOK) is the 
complete set of concepts, terms and activities that make up a professional domain. The literature 
describes the purpose of bodies of knowledge in their role as a requirement for professional 
identity, the development and furtherment of professionalism, and the acclamation and 
dissemination of knowledge to embody shared values and practices. Professional status requires 
the field to monopolize a discrete BOK, but how this knowledge is organized varies from 
profession to profession which may present difficulties in the development of conceptual 
frameworks for continuing education course development where courses are frequently created 
and aligned to developing trends and issues in the profession. 


The world and cultures in which we live, and experience are constantly changing. 
Continuing education course work and certificate programs allow workers to stay current with 
the latest developments, knowledge, skills, and technologies of their perspective fields. Some 
professions also require the completion of continuing education coursework to comply with laws 
governing licensing and certification within the profession. In the creation of continuing 
education programs, as in the construction of any coursework or program, most instructional 
design models begin with an evaluation of the structure of the knowledge and skills. Structure is 
determined and instructional content, activities, and assessments are developed. “This may seem 
quite straight forward, but the designer faces a problem: the different instructional design 
theories do not show much similarity in the way they describe ‘knowledge and skills. Sometimes 
even the labels ‘knowledge’ and ‘skills’ are not explicitly used” (Dijkstra, 1991). More still, the 
identification of knowledge and skills for subject matter experts in the development of 
coursework and programs may be hampered by the limitations of their own evolving knowledge 
of practice within the field, and the expectations of course development at the institutional or 
organizational level. The successful identification of knowledge and skills upon which any 
coursework and programs will be based is the first step in developing successful continuing 
education programs. An identified BOK and its organization and resulting skill development 
may inform the selection of an approach for course and program design and more information 
about how knowledge is organized should be a part of the instructional design process. 


Continuing Education Programs 


The development of continuing education (CE) programs, which for the purposes of this 
paper will be considered workforce learning outside of traditional degree programs and trade 
schools, was born from a demand by enterprise as early as World War II for “employees to be 
knowledgeable, skilled, and responsive to social and professional changes” to enable employers 
to compete in the marketplace (Mizzi, et al, 2020). The public policies of the Roosevelt era 
emphasized workforce education and training would be necessary for Americans to join the new 
developing economy (Roumell & Martin, 2020). Public policy continued to herald this need. In 
2014, the Workforce Innovation and Opportunity Act reinforced the purpose of continuing 
education programs as an opportunity for access to education and training needed to be 
successful in the labor market. 


At the same time professional organizations during the 20" century and to this day have 
spent time in the development of “boundaries” surrounding their professional practice. Most 
professional organizations determine what formal education and entry requirements are 
necessary for the profession and exercise autonomy over the conditions and terms of ethical 
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practice; and, with this process, the organization develops a monopoly over a discrete body of 
knowledge and related skills (Morris, et al, 2006). How each organization disseminates that 
information among its membership, including education and training, which results in continued 
membership, certification, and licensure varies from profession to profession and, sometimes, by 
level of government oversight. 


Subject Matter Experts 


When attempting to serve the profession through education and training, educational 
providers seek to access the BOK of the profession frequently through subject matter experts 
(SMEs). What information these SMEs provide about the BOK of any given profession varies 
according to the skills, knowledge, and training they possess. In seeking to develop courses or 
programs for continuing education, instructional designers may be tasked with determining an 
organization for the knowledge presented by the SME. At this time, there is no systematic 
process or conceptual framework for evaluating a BOK presented by a SME. This circumstance 
may present a challenge to organize the presented knowledge effectively for learning design. 
Further, the perspective from which a SME may deliver a given BOK for course development 
may also affect the quality, type, and depth of knowledge received by the instructional designer 
further complicating the process of knowledge organization. 


Morris, et al (2006) expressed concern about the focus and function of SMEs in the 
effective delivery of knowledge in their explorations of the BOK associated with project 
management. The authors noted that from the “socially constructed viewpoint of knowledge, 
one cannot avoid reflecting on the power relations” of the actors on a BOK. In Figure 1, the 
matrix of SMEs that serve as professional knowledge aggregates is an interpretation of these 
roles. 


Figure 1 


Subject Matter Experts as Professional Knowledge Aggregates 


Subject Matter Primary Function Area of Focus Contribution to Influences on 
Experts Bok Behavior 


Transfer knowledge to 
Convert abstracted readily comprehended 
knowledge into a saleable and generalized to a cs = 
& applicable form variety of disparate 
contexts 


Translate knowledge into Legitimize practices 
practice forms that can through presentation 
be understood & used by at conference & 
practitioners, promote practice 
consultants, & trainers through consulting 


Professional 
Association 


Funder of consulting Drive the 
services & training improvement of 
— requiring standardization practice to update 
& certification for qualifications and 
employment lifelong learning 


Academics Validation or quality 
& control by examining and 
critiquing practice 


Researchers through research 


Actively control the 
required education 
process 
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In their case study, these authors recognize the complexities and different functions that 
associated SMEs have in relation to the BOK (Morris, et al, 2006). The professional organization 
serves as the coordinator of the collective group of SMEs exercising regulation of the community 
of practice through registering participants and providers, sponsoring conferences, and funding 
research. It is common practice among professional organizations to view the identification of a 
BOK to which the organization will subscribe as a steppingstone in unifying the community of 
practice. 


Within the professional association, consultants and gurus find their opportunities to 
affect the BOK through participation in the organization, though their purposes, focus, and 
function are different. These motivations accompanied with other influences on their behavior 
affect their contribution to the BOK. Consultants ensure that knowledge is easily comprehended 
and generalized for the community of practice and those attempting to join the profession while 
gurus legitimize practices and demonstrate expertise. Combined with the function and focus of 
enterprise and government serving as consumers, a provider/consumer relationship is recognized 
within the community established by the professional organization. Academics and researchers 
provide the quality control and validation for the BOK, which affords control of the educational 
processes of the profession. Morris, et al (2006) also acknowledged that there was a balance 
struck among the SMEs that sustained the BOK within the community of practice. 


A well-established professional organization with an intent on regulating or establishing 
the boundaries of professional practice is positioned to provide governance and regulation to the 
development of the BOK. Standards of Practice are often the result of this governance and 
regulation. It can be equally effective for instructional designers in the role of organizing 
knowledge provided by a SME for course development to recognize the SME’s role in the 
profession and therefore their approach to the BOK as well. Their approach to defining the BOK 
for a given profession may provide insight into the assessment of given knowledge and its 
organization. 


Categorizing Knowledge for Organization 


How we define a BOK varies according to who is defining. For the academic, a BOK is 
the complete set of concepts, terms and activities that make up a professional domain, its 
ontology. According to Oren (2005), it is the “structured knowledge that is used by members of 
a discipline to guide their practice or work.” Dijkstra (1991) acknowledged that “there are 
differences in the ways that the description of knowledge and skills for purposes of instruction 
can be described” but also acknowledged that “these differences were not much help to 
instructional designers”. To define knowledge organization for instruction, Dijkstra borrows 
labels for knowledge types from cognitive psychology. Knowledge is broken down into three 
types--conceptual knowledge, causal knowledge, and meta knowledge—and associates a 
problem as an interrogative statement and connects relevant skills to the knowledge type, as seen 
in Table 1. 
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Table 1 


Dijkstra’s (1991) Types of Knowledge, Related Problems, and Relevant Skills 


Knowledge Type Problem Relevant Skills 

Conceptual Knowledge 

Fact What is the name of this Recognition 
single symbol, object, event? 

Concept 


Class concept 


Relational concept 


Causal Knowledge 
Conditions and biconditions 
(series of events, process, 
causal chain) 


Meta Knowledge 


Plans, Strategies 


To which category does this 
object belong? 

What is the relationship 
between these objects? 


What will happen after a 
certain time lapse? 


How to plan, how to attack a 
problem? 


Identification operations 
--Categorization 
--Application of problem- 
solving procedures 


Making predictions by 
application of the lawful 
relationships 


Thinking skills, self- 
regulatory skills 


Anderson and Krathwohl (2001) categorized knowledge, separating facts from concepts into 
factual knowledge and conceptual knowledge staying abreast of changes in cognitive 


psychology, as seen in Table 2. 


Table 2 


Anderson and Krathwohl’s Knowledge Dimensions 


Factual Knowledge 


Essential facts, terminology, details to understand a 
discipline or to solve problems within it. 


Conceptual Knowledge 


Classifications, principles, generalizations, theories, 
models, or structures that enable function. 


Procedural Knowledge 


Information or knowledge, methods of inquiry, specific 


skills, algorithms, techniques, and particular methodologies 


establishing criteria for action. 


Metacognitive Knowledge 


Reflective knowledge about how to go about solving 
problems, cognitive tasks, including contextual and 
conditional knowledge, thinking about the thinking of 


practice. 
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It is when we combine the SME function, focus, and contribution as knowledge aggregates with 
the four dimensions of knowledge, as seen in Figure 2, that we may have a method for 
categorizing the knowledge presented by SMEs for continuing education program development. 


Figure 2 


Subject Matter Expert Primary Function and Association Knowledge Dimension 


Subject Matter Primary Function Area of Focus Contribution to BoK Knowledge Dimensions 
Experts 


Transfer knowledge to 
Convert abstracted readily comprehended 
knowledge into a saleable || and generalized to a 
& applicable form variety of disparate 

contexts 
Translate knowledge into Legitimize practices 
practice forms that can through presentation 
be understood & used by at conference & 
practitioners, promote practice 
consultants, & trainers through consulting 
Funder of consulting Drive the 
services & training improvement of 
requiring standardization practice to update 
& certification for qualifications and 
employment lifelong learning 
Academics ean cua Actively control the 
& one : eee required education 
Researchers Smiaas Peake process 
through research 


If we consider the role from which a SME may approach the profession and the knowledge 
dimension they provide within the BOK, instructional designers may develop approaches to 
knowledge organization to inform course development for continuing education programs that 
more closely align course goals with desired outcomes. In this way instructional designers also 
acknowledge that SMEs do not demonstrate command of the body of knowledge with the same 
focus or contribution. Understanding what a SME may contribute also supports our 
understanding of the role of knowledge aggregates defining the boundaries of the profession. 


Approaches for Developing Programs from BOK Knowledge Organization 


With the understanding of the SME’s contribution and what knowledge dimension is 
represented by their work, an approach to program planning may be made. Each approach 
focuses on the contribution of a specific SME and how that knowledge may be developed into a 
context for learning. Examples of suggested approaches are provided. 


Approach 1: Foundational, Introduction to Profession 


Knowledge is curated concepts and nomenclature, essential facts, terminology, details to 
understand a discipline or to solve problems within it, as most often created by consultants. 


Example. Most continuing education courses found on Linkedin’s Learning platform. 
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Approach 2: Role of the User, Organization or Project 


From the field of practice, conceptual knowledge consisting of best practices provide the 
framework for the program of study or analysis of case studies to identify relevant professional 
skills (Romme, 2016). 


Example. Web Accessibility Certificate 
Approach 3: Levels of Implementation 


This approach is developed from procedural knowledge and the most used for 
professional continuing education courses associated with the licensing of service providers. This 
information can be drawn from professional organizations (INFORMS, 2009). 


Example. Project Management Professional (PMP) 
Approach 4: Levels of Performance 


This approach is most often used in technical trades, generally providing a three-part 
advancing model (Oliver 2012), which incorporates factual knowledge, conceptual knowledge, 
and procedural knowledge. 


Example. Beauty School/ Real Estate Certificate 
Approach 5: Competency Construction (Modified Degree Model) 


Developed similarly to degree construction but stripped down to identified competencies 
based on meta knowledge for shortened time experience while focused skill acquisition 
development. 


Example. Teaching Certificate Redesign: Making a Flexible Program for Future Faculty 
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Abstract 


This paper describes a series of instructional design projects focused on the design and 
development of successive pregnancy and postpartum fitness courses to provide continuing 
education courses for sets of learners with specific identities and learning goals and 
objectives. Taken as a series of decisions and actions employing the modular rapid prototyping 
techniques borrowed from Atomic Design and the collective decision making of Agile 
development, a design object-oriented approach to rapidly produce an array of continuing 
education courses is described. In this context, an instructional design team was able to redesign 
learning objects to meet the criteria of each specific learner identity, or “User Story.” This design 
case presents the critical design decisions made during each course development iteration, the 
reasons for those decisions, failures in which the design did not work as planned, and reflections 
on the design and development process to establish the precedent revealed in a new instructional 
design model, which when replicated will result in newly designed courses. 
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Background 


Continuing Education (CE) course design is an area of Instructional Design that requires 
constant modification and technological implementation to meet the competitive demands of the 
global marketplace. The ability to produce CE course content in a timely manner is a key factor 
in creating sustainable revenue for any institution. According to Adnan and Ritzhaupt (2018), 
there is an increasing demand for instructional designers to create high-quality instructional 
solutions in a variety of settings using more effective approaches. Adnan and Ritzhaupt suggest 
that the integration of software design principles into Instructional Design processes are a 
pragmatic approach that can make the creation of courses more productive and efficient. 

Borrowing from manufacturing processes, one technique of interest for producing 
essential and accessible content in a fast-paced environment is Rapid Prototyping (RP). Through 
modular design and employing the Successive Approximation Model (SAM), developed by 
Michael Allen, instructional designers have been able to move away from the linear, time- 
consuming, waterfall design processes and quickly and cost-effectively produce content 
(Desrosier, 2011; Sites, Allen, & Green, 2014). Using this technique allows iterative design 
practices to drive the creation of a fast and more cost-effective production of course and content. 
RP can be practiced with AGILE project management principles that allow for increased 
communication between the stakeholders, contributing to a more successful delivery of the 
product. In small institutions, CE course development may be more effective with the 
implementation of AGILE design principles because of the constantly changing course demand 
and the need for quick turnaround time for courses under contract. 

In this design case, we will describe how one course development process considered the 
development of user stories to produce multiple iterations of the course content, tailored to the 
needs of each user. This process decreased the production time of CE courses and produced an 
array of fully developed course products for multiple audiences. 


Institutional Context 


This design case details the experience of the initiation of a CE course by a Subject 
Matter Expert (SME), the design and creation process, implementation, and further iterations of 
the original course. The layout for the design process and the rationale for the use of the 
Successive Approximation Model (SAM) in this process are explained. The outcomes and 
reflection process are shared to reveal a series of decisions that led to the development of a new 
design model — the User Story Approximation Model (USAM) and the implementation of a best 
practice methodology. 

The university that employed the designers of this case is a fully online degree-awarding 
institution that also offers CE courses to professionals in the field of sport. The CE department 
provides certificates and courses for domestic students as well as international partners. Most of 
the contracts for CE are with international organizations and institutions so the timeline for 
course development is much shorter and the design more situation-specific than that of domestic 
continuing education. 

The university’s Instructional Design department had previously used a linear, waterfall 
approach to project management and course design. This process created lengthy design times, 
difficult communication, and course revisions. Using the linear ADDIE approach, course 
production covered months of development time, while content moved from one person to 
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another. The department underwent a change in management and the new director shifted this 
process and implemented Agile principles and a Scrum methodology. The department then 
began to use the SAM model to develop course components quickly, which decreased production 
time substantially. Designers completing other courses found utility in the reuse of components 
from this course. It served the department’s functioning to make the course components a part of 
design element library that was collectively used for decreasing the amount of design time and 
creation while building courses. 

Using the SAM model for iterative development, we were able to work with modular 
units and standardized design elements to provide a flexible framework for all continuing 
education courses. This framework was easily accessed by the team of designers, managers, and 
SMEs to rapidly create and edit course content. The design element library was housed on a 
shared drive in which all designers accessed. Once the initial course had been created, the 
modular components of design and content were then used to create multiple course iterations to 
meet the demands of other programs and product owners. This systematic approach to course 
design based on atomic design principles from software design practices allowed a small team to 
produce more courses in a shorter period (Frost, 2016). 


First Iteration (Alpha Course) 


The development of the Alpha course began with a Subject Matter Expert (SME), who 
was a graduate student and fitness professional who was contracted by the university to develop 
the concept of a maternity wellness course that incorporated personal fitness exercises to create 
for domestic CE course. This course was intended to serve as an informative progression through 
pregnancy to aid women and interested individuals in identifying the transformative changes and 
safe exercises during pregnancy and the postpartum period. The SME created an outline for the 
course and provided a plan for instructional videos that she would be recording to demonstrate 
the exercises. The instructional designer met with the SME. The course objectives and desired 
outcomes, flow of the modules, existing media, and resources were determined for the course. 
The SME agreed to write content for the unit pages. The course modules were developed in a 
branded format. The course elements were added to pages and content was added in a successive 
fashion as the SME provided it. 

The design for the course consisted of a five-module structure, including an introduction, 
content pages, reading material, auto-graded knowledge checks, and video resources in each 
module. The modules corresponded with each of the three trimesters, postpartum fitness, and 
associated resources. The modules included content pages with a title that described the key 
concepts found on the page. The course detailed the safety of exercise and the changes of the 
human body during and after pregnancy. The SME provided specialized content of an interview 
that she conducted with a professional in the medical field, which was included in the first unit. 
The instructional designer filmed and edited the instructional videos in the university’s recording 
studio, and the videos were added to the modules. The instructional designer also located an 
actor with a baby to demonstrate postpartum exercises for video production as well. 

The course was completed in one month, from design to creation, and then moved to sale 
on the university’s CE website. Once the course was live for enrollment, the marketing 
department advertised the course to increase enrollment and interest in the course. The visibility 
of the course to multiple university partners attracted interest in modified versions of the course. 
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Successive Iterations 


An inquiry was made by a Global South institution about creating a Spanish version of 
the course. The ID team met with the stakeholder to discuss the existing course and possibilities 
of translating the course to Spanish. The stakeholders provided documents describing content 
that they would also like to see implemented into the course. The scrum process that was used by 
the ID team. A retrospective identified deficiencies in the Alpha course for the new iteration. 
Goals were set so that a backlog of the tasks could be created to establish design priorities and 
work was completed through a task to time assignment arrangement called a sprint. The ID team 
completed the revision of the Alpha course through an implementation of Weick’s sensemaking 
to orient tasks to the new user story established by the stakeholder. Content was expanded and 
new media was created and added to meet the demand of the user story. The stakeholder 
reviewed the course and provided translation documents to orient the course to the Global South 
partner’s dialect. At this late stage, the stakeholder sought renegotiation to include branding 
images within the course content. Once these items were added to the course, the course then 
moved from development to sales and marketing on the university’s CE website. 

It was determined by sales to offer the Alpha course as a complimentary course on the 
university CE webpage to highlight new multimedia implementation for internal stakeholders 
and to offer a valuable resource to the local community. This complimentary iteration of the 
course supports a portion of the university’s mission to provide free education to the public. The 
first iteration in English became a course for domestic sales and the Spanish translated iteration 
became a sales offering to Global South partners. 

The ID team was then notified about an inquiry from a US partner institution that wanted 
to purchase a course that provided training in pregnancy and postpartum fitness for nursing 
students. The introduction of this new user story for the course sparked the next scrum meeting 
for the ID team to identify other user stories that could benefit from a course in pregnancy and 
postpartum fitness. The ID team identified the target audiences, their individual needs and 
developed multiple user stories for iterations of this course. At the end of the planning meeting, a 
total of five courses were identified with separate user stories (audiences). The user stories 
determined the successive iterations of the Alpha course each time implementing Weick’s 
sensemaking to meet goals and outcomes established by the user stories. 

The content expanded after developing learning outcomes for each course. Outlines for 
each new pregnancy and postpartum fitness course with components aligned to the user stories 
were created for delivery to various institutional clients in tandem development processes. The 
ID team’s rapid prototyping through repurposing learning and design objects and AGILE 
collaboration on the course design led to production ready courses within two days. 

The process began with the duplication of the Alpha course into successive course shells 
and labeled for each new user story. Page elements from the design library were added to ensure 
accessibility, brand, and consistency. The ID team then edited the course content through 
Weick’s sensemaking. The content was considered and then modified through four critical 
processes of interpreting the content for the new user, transmuting the structural elements, 
providing variations to focus on the user’s needs, and finally improvising any missing content 
that would complete the learning goals and outcomes. The instructional designers created or 
modified media and graphics for each successive iteration. Media was created with reusable 
design elements that provide a standardized and branded look, while making production time 
minimal. Once the courses were complete, the primary shells were duplicated, and each given a 
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unique course name and number to identify the audience. The SME was then asked to review each 
iteration to make modifications to content according to audience and to the interactive learning 
components and assessments to meet the specialized interest of the stakeholders, thereby producing 
the final version of the six courses, which were delivered within a two-week period. 


Reflection on the Process 


The ID team realized that they had created a series of decisions that through their 
documentation would create a set of best practices for rapid design and development. Creating an 
array of CE courses through the identification of user stories served as a benchmark that led to 
the development of an instructional design model. The use of AGILE practices allowed for 
collaboration that eliminated the need to pass content in a linear fashion, cutting down on the 
time needed for creation and revision, and allowing for multiple iterations of courses to be 
identified and created. The final retrospective of this development cycle led to plans on how to 
improve the quality and effectiveness of the new design process. The ID team found a common 
feeling of motivation from collaboration and communication. Collaboratively processing course 
design at the same time, using video conferencing as the work was being completed, allowed for 
decisions to be made in real time. The use of modular design and a design element library 
allowed for quality content to be produced and branded in an efficient and lucrative manner. This 
set of process helped define a best practice for converting alpha course content into multiple 
iterations. The feedback from the stakeholders was also incredibly positive and in the case of one 
stakeholder, additional course contracts were negotiated for other content areas. The process of 
RP of course content has shown to be cost-effective and essential for a small institution with 
limited staff and budget. 
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Abstract 


This session includes justification for personalized/customized learning assignments in higher 
education; some benefits of using this approach; and a discussion of how such assignments 
promote critical thinking skills as students work within the higher levels of Bloom's Taxonomy. 
Low-tech, mid-tech, and higher-tech differentiation strategies are discussed via examples of 
these types of assignments that the presenters use in their own classes. 


Introduction 


Differentiated learning. Personalized learning. Choose-your-own adventure. Choice 
boards. These concepts have long been discussed and implemented in the K-12 environment 
(Coppens, 2021), but their adoption in higher education has been slower. 

Why should professors implement customized or personalized assignments in their 
classrooms? The reasons are many. Most people like talking about themselves and enjoy 
integrating their experiences into assignments. Students appreciate relating their schoolwork to 
something practical. Instead of wondering when they will use an assignment in “real life,” 
students find themselves using, applying, and making meaning of topics immediately. 
Personalized assignments give students ownership of their work, cut down on cheating, and 
encourage students to perform at the higher levels of Bloom’s Taxonomy (Armstrong, 2010). 
Instead of working within purely knowledge-based assignments, personalized assignments can 
help students produce artifacts in the analysis, synthesis, and evaluation spheres. 


What can (or should) be personalized? 


Personalizing assignments can initially feel overwhelming. A professor might wonder, 
“Am I supposed to make a unique assignment for every student in my class?” Instead of the onus 
for personalization falling solely on the teacher, however, consider the role of the student in 
personalization. The students may have great ideas about how to personalize assignments to their 
current jobs, classes, or future careers. Remember that not every assignment needs to be 
personalized. Some assignments should be completed word-for-word, step-by-step by every 
student so that every person has the same basic understanding of introductory material. Such 
knowledge-level activities, designed to help students remember content, are at the lower levels of 
Bloom’s Taxonomy and may not lend themselves as well with personalization. Often, students 
must pick up the background knowledge about a subject before they have the big-picture 
perspective to be able to personalize anything. Admittedly, some subjects or projects may lend 
themselves better to personalization than others. Professors must embrace the discomfort of 
having students submit assignments that may look different than what the professor envisioned. 
Often students may submit assignments that are better than what the professor envisioned! 
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Having a high-quality rubric for any assignment that a student can personalize helps to reduce 
(unwanted or poor-quality) surprises with the end product. 

Professors tend to approach differentiation in one of three ways—they may disregard it, 
adapt to it, or transform the process (Boelens et al., 2018). Professors who do not offer any 
customization in their assignments would fit under the disregard category, and professors who 
allow a great deal of differentiation/customization in their assignments would fall under the 
transform category. While neither end of the spectrum is “right” or “wrong,” it is important to 
self-reflect and think about which end of the spectrum one currently falls. 


Ways to Personalize 


Professors must first decide, for a given project, if the format or the content is the most 
important piece for the students to master. If one matters much less than the other, then that is the 
area that may be best for personalization. For example, if a teacher’s goal is for a student to learn 
a specific technology—say, Flipgrid—tthen the format matters, but the content (what the student 
discusses in his/her Flipgrid) doesn’t. This professor can give the students agency in the topic, 
style, and presentation of the Flipgrid. A biology professor may want his students to learn the 
basics of mitosis—the content is non-negotiable—but the format in which the student relays the 
information about mitosis can vary from student to student. Modifying content may not always 
be possible due to industry standards, but professors can consider ways to modify their process 
and product to the benefit of the students (Pham, 2012). 

Simple ways to personalize assignments are as follows. Professors can offer choice 
boards. These boards are like tic-tac-toe boards, and students must complete X number of 
assignments, of their choice, to submit for grading. For example, an instructor could allow a 
student to demonstrate mastery of a topic through writing a poem, drawing a poster, creating a 
PowerPoint, writing an essay, filming a video, performing a skit, making a brochure, making a 
children’s book, etc. The student could choose three of these activities (each of which has a well- 
developed rubric) and submit the artifacts that s/he finds most relevant, interesting, or relatable. 
Even on a single assignment, such as a paper or discussion board, students can choose a prompt 
to which to respond, offer their own prompts, or include a section about how the subject material 
relates to their lives or future/current careers. Campbell and Cox (2018, p. 11) noted that “digital 
video was an authentic and personalized learning experience that fostered personal choice and 
voice and peer collaboration.” When students have agency in their assignments’ parameters, they 
feel more competent, autonomous, and related to the group (Danley & Williams, 2020). When 
preservice teachers have professors who provided differentiated learning experiences, they 
themselves are more likely to differentiate to their own students when they enter their own 
classrooms (Joseph et al., 2013). Marghitan et al. (2016) note that students’ intrinsic motivation 
and final grades increase when the students are offered choices in workshop and lab 
opportunities. 


Examples 
The authors of this paper both teach primarily graduate-level students in instructional 
technology classes. Many of their classes are project-based, and the majority of the students 


work full-time and attend school part-time. It is the goal of the professors to have the students 
create artifacts that demonstrate specific competencies that they can use in their daily jobs as 
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opposed to assigning “busy work.” Here are some specific examples of assignments they use in 
their classes that allow for customization or personalization from the students. 


EFD 552: Diversity in the Classroom 


This class helps current and aspiring teachers understand the backgrounds, needs, and 
strengths of children from diverse upbringings. Students create a Schoolwide Diversity plan for 
their capstone requirement. In this project they use findings from scholarly research along with 
data from the school at which they work to identify underachieving student subgroups and 
present ways their school can better address these student needs. They author a paper, create a 
PowerPoint presentation, and record a screencast of them presenting their ideas to stakeholders 
where they discuss their ideas. They are encouraged to share their findings with principals, 
fellow teachers, parents, school boards, and other stakeholders so that their carefully researched 
suggestions might be considered for implementation locally, leading to positive change at their 
schools. 


EIM 504: Learning Through Interactive Technologies 


This class is project-based, and students experiment with different emerging or 
established educational technologies. Each technology learned is then related by the student to a 
standard they teach, or an ISTE standard. Since students have varying proficiency levels, more 
advanced students are allowed to explore technologies not covered in the class so that they are 
still learning new things. Students are encouraged to incorporate their certification field, grade 
levels, and school policies into all artifacts. 


EIM 505: Digital Literacy in the Classroom 


In this class, current and aspiring teachers learn how to evaluate digital sources, 
implement them into their classrooms, and teach their own students how to think critically. 
Students “choose their own textbook” for the class from a list of popular nonfiction books. They 
select an Alabama Virtual Library database of choice to teach to classmates, and they curate 
digital literacy resources in a class wiki and post to a shared class blog. Each student brings 
his/her own knowledge and expertise to these tasks and takes ownership for passing that 
knowledge along to other class members. 


EIM 517: Designing Virtual Learning Spaces 


Students in this class design, built, and deploy online courses in a learning management 
system. They get to choose their topics, subjects, grade levels, etc. For those who are current 
teachers, they can use existing assignments, or they can use this class as an opportunity to also 
create new assignments more suited for the virtual or blended environment. Ideally, the students 
will take what they created in the class and deploy it in the near future to “live” students, 
especially relevant in the era of pandemic teaching. 
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EIM 555: Instructional Design 


This class covers high-quality design standards, whether in-person or online. Students 
pick a passion project unrelated to K-12 teaching and design, from the ground up, a complete 
mini-class on the topic. Students produce Gantt/PERT charts, infographics, rubrics, lesson plans, 
objectives, needs/task/learner analyses, and more. By having the students pick a fun, “non- 
school” topic, they can remove preconceived notions about lesson planning and focus on creating 
a high-quality instructional project based on something they love, such as travel, baseball, or 
gardening. 


EIM 610: Emerging Technology and Collaborative Tools 


This class is for students earning their educational specialist degrees. All students are 
current fulltime teachers across various disciplines. They research podcasts, assemble social 
bookmarks, and create YouTube channels about their discipline and share their findings with the 
class. They learn to differentiate among low-, mid-, and high-tech solutions in their discipline, 
and they train other students about technologies specific to their discipline. 


UH 101: Priming Students for Study Abroad 


This class is taught to college freshmen and sophomores and is intended to support first- 
generation college students, or those who have traveled little, in their quest to become study 
abroad participants. During the class, students plan/budget domestic and international trips. They 
complete a study abroad application and a study abroad scholarship application based on their 
ideal study abroad location. They choose 2-4 activities offered around campus that get 
themselves out of their comfort zone (such as trying a cardio fitness class, attending a lecture on 
a topic they wouldn’t normally find interesting, or taking an Uber); complete these activities; and 
reflect upon them. 


General Ideas for Personalization/Customization/Extension 


In any class, students who author exceptional papers can be encouraged to submit them 
for publication. This leads them to investigate journals in their field and become more involved 
professionally. With interview assignments, allow students to pick their own interviewees and 
craft their own questions. Consider a statement in syllabi to the effect of, “If you have a better 
idea/way to approach/implement this, suggest it for approval!” If students in your classes are 
already experts on a specific unit, engage their help in teaching the unit, tutoring others, or 
demonstrating practical applications of the topic. In some classes, students who already possess 
mastery of a unit can design their own alternative learning experience and submit it for approval. 


Roundtable Questions 
The following guiding questions will serve as a starting point to promote discussion at the 
roundtable. The session will be designed to solicit input and engagement from the attendees, as 


we value their input and look forward to hearing ideas about how they customize or personalize 
assignments in their own classrooms. The roundtable will be a springboard of ideas and also 
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serve as a "genius hour" of sorts as like-minded professors share personalization strategies that 
they use in their own classes. 
1. Please share with the group any ideas you have about how to customize or personalize 
class assignments. 
2. What classes or subjects do you think work best with customized assignments? What 
classes do you feel would be difficult to allow personalization in assignments? 
3. What pros or cons do you feel exist with this style of teaching and assessment? 
4. Brainstorm ways that a professor could offer personalized or customized assignments in 
the “difficult” subjects. 
5. Brainstorm how you can give fewer multiple-choice tests in favor of more authentic, 
personalized assignments. 


Conclusion 


Although examining the literature regarding customized and personalized learning is 
useful, it is also very beneficial to have face-to-face discussions with professionals in the field. 
Learning what others do well, and then seeking to do it oneself, is a way that professors can 
increase their scholarship of teaching. Using guided questions, this roundtable will provide a 
forum for these conversations. 
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Abstract 


In this case study of a course on Cybersecurity and Biometrics, we explore the 
effectiveness of project-based learning using a makerspace combined with the 5E Instructional 
Model. We describe the course design and its objectives, the project itself, and the makerspace 
facility where students undertook the project. Using a pre- and post-project student survey, we 
examine changes in student perceptions over the course of the project. We uncover some 
interesting insights relative to the use of a project as a learning tool in technical courses such as 
this, particularly regarding pedagogy, teamwork, incorporation of a makerspace, and online 
learning. 


Section 1 - Introduction 


A Cybersecurity course is challenging to teach and learn. Bridging the gap between 
theory and practice is problematic due to its abstract and highly technical content, and online 
teaching adds to this complexity. With this in mind, we conducted an experiment using a 
makerspace for experiential learning in conjunction with a project-based approach and the SE 
Instructional Model to bring theoretical concepts in a Cybersecurity Biometrics course to 
practical implementation. 

To study the effectiveness of our experiment, we used a case study approach where 35 
students in an online Cybersecurity Biometrics class were grouped into teams for a semester- 
long project to integrate theory and practice by building a working prototype of an optical 
fingerprint reader using a Raspberry Pi, an Internet of Things (IoT) camera, a glass prism, the 
National Institute of Standards and Technology (NIST) Biometric Image Software (NBIS) 
modules, a database, and a 3D printed case to integrate these components. Each team followed 
project management processes and designed their own 3D case, database and program interface. 
The makerspace provided students the opportunity for ideation, prototyping, learning from 
failure, and creativity. Pre and Post surveys, employing a modified 7-point Likert Scale ranging 
from Strongly Disagree to Strongly Agree, were used to explore our research questions and 
assess the effectiveness of a makerspace project-based experiential learning to teach 
cybersecurity and biometrics concepts through an online modality. 
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In section 2 of our paper we review principles and related work on the 5E Instructional 
Model, project-based learning, and the utilization of makerspaces as a learning tool. Section 3 
describes the methods and details our approach with SE, project-based learning and structure, 
technical specs and biometric concepts of an optical fingerprint scanner, makerspace lab 
resources, and pre/post-survey sections. Pre and post survey data of key results and analysis is 
discussed in section 4 and section 5 contains a brief conclusion of our study. 


Section 2 - Related Work / Literature Review 


Instructional models are effective at helping students learn STEM concepts through 
inquiry and experimentation (Bybee, et al., 2006). The 5E instructional model consists of 
Engagement from prior experiences, Exploration through inquiry of past knowledge, Explanation 
of what the student understands, Elaboration of understanding through new experiences, and 
Evaluation of the student’s own understanding. The BSCS 5E Instructional Model was 
developed in the late 1980s and have been used broadly and effectively across multidisciplinary 
STEM curriculums and professional development (Bybee, et al., 2006). 

Research and case studies on the utility and implementation of the 5E model have been 
shown to be effective in many disciplines and settings such as teaching Physics and Newtonian 
mechanics to over 150 undergraduate students (Bahtaji, 2021); at the elementary and middle 
school levels to teach STEM, buoyancy force problems (Cepni & Sahin, 2012), mathematical 
modeling of geometric objects (Tezer & Cumhur, 2017), or exploration of concepts and solutions 
of STEM problems affecting local communities (Bybee, Using the BSCS 5E Instructional Model 
to Introduce STEM Disciplines, 2019); online hybrid teaching modality for Special Education 
Teachers (van Garderen, Decker, Juergensen, & Abdelnaby, 2020); and at the graduate level in a 
Network Security course (Olimid, 2019). Although the latter represents an example of a 
successful implementation of the SE Instructional Model for teaching Cybersecurity, this is a rare 
case study because the 5E Model has not been widely researched or applied in teaching 
Cybersecurity shown by the limited published research. 

Project-based Learning provides a structure, processes, and workflow for students to gain 
deeper knowledge through creative and critical thinking, and manage uncertainty with real-world 
problems through communication and teamwork while learning professional skills (Anazifa & 
Djukri, 2017; Johnson, 2019). This approach is commonly used in teaching Cybersecurity. It 
keeps students engaged, focused on specific tasks, and directs the learning process through the 
development of diverse solutions to real-world problems (Sherman, et al., 2019). 

Makerspaces are venues for carrying out do-it-yourself (DIY) activities. They typically 
are home to various machines and hand tools, including 3-D printers, laser cutters, vinyl cutters, 
vacu-formers, sewing and embroidery machines, among others, and laptop or desktop computers 
with design software. In the education realm, they are used to give students the opportunity to 
engage in ideation and to build prototypes, as part of a project-based learning component of a 
given course. As of 2020, there were over 2,300 makerspaces globally (hackerspaces.org, 2021). 

Makerspaces are the physical manifestation of the maker movement, considered by some 
to be the most important economic movement since the Industrial Revolution (Anderson, 2012). 
It represents a migration away from mass production in factory facilities to small-scale 
manufacturing often by end users. Makerspaces have been found to promote creativity and 
innovation and makerspace projects frequently lead to new business startup (Halbinger, 2020). 
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Section 3 - Methods 
Section 3.1 - 5E Instructional Model 


The 5E instructional model was used an online asynchronous Cybersecurity Biometrics 
class. Lectures on fundamentals of biometrics were delivered through Zoom sessions. These 
lectures were used to engage the students through discussions of project management, computer 
science and cybersecurity topics such as teamwork and communication; the security triad of 
Confidentiality, Integrity, and Availability (CIA); Authentication vs Authorization; Identification 
vs Verification; programming languages/libraries; and databases. These discussions also 
provided an opportunity for the students to explore their understanding of computer science and 
cybersecurity concepts from previous classes and how these can be applied to the assigned 
project. At the end of the semester students were required to produce a written report with 
analysis of their project results and provided a presentation to the class. This process allowed 
them to explain their understanding of the cybersecurity and biometric concepts and elaborate on 
their newly acquired insights based on the analysis of the project results. Evaluation of the 
student’s own understanding occurred as other groups presented their report and analysis of their 
project and the students were able to compare and contrast their personal understanding with that 
presented by their classmates. 


Section 3.2 - Project-based Learning and Teaching 


In this case study, students were instructed to form groups of four or five and were 
provided detailed project instructions to create a structured project approach. Instructions 
included coming up with a team name, selecting a team leader, identifying student’s technical 
skills, defining roles and responsibilities, creating a schedule of tasks and timelines, and agreeing 
on a communication plan and meeting frequency. 

Defining a team name to help enhanced their perception of identity and belonging to a 
group. Students approached this project with a competitive spirit; this was an unexpected side 
effect observed through student comments throughout the semester and during the presentations. 
They also showed pride in their project solution during the presentation by sharing personal 
experiences on how they overcame specific challenges. 

The selection of a team leader is an important step to define roles and responsibilities and 
create an organizational structure to manage the project. The team leader managed project tasks 
schedule and communicated with the team and the stakeholder (instructor). Team members 
helped define the expected timeline and effort to foster a sense of responsibility and ownership. 
Team leaders were tasked with identifying each team member’s technical skills and defining 
their roles and responsibilities. This is a critical step in the team formation to ensure everyone in 
the team understands their contribution and what the team expected of them. This approach was 
derived from the organizational development literature on team building, and particularly the 
work of Richard Beckhard (Beckhard, Optimizing team building effects, 1972) (Beckhard & 
Harris, Organizational transition: Managing complex change, 1987). Beckhard was the co- 
developer of a model for team building called GRPI (goals, roles, processes, and interpersonal 
relationships), which he asserted were the keys to successful and high-performing teams. Team 
leaders met biweekly with the instructor for mentoring on technical challenges, team dynamics, 
and share ideas and provided a written progress status. 
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Section 3.3 - Cybersecurity, Biometrics, and Optical Fingerprint Reader 


Each team was instructed to design, develop, and implement a Biometric System for an 
optical fingerprint reader. Students were introduced to the concepts of biometrics and key 
factors that makes a biometric system functional and secured through a series of online lectures. 
Students were provided a project description to guide them through a set of requirements and 
asked to be creative and design a working prototype that is technical, functional, and secured. 

The National Institute of Standards and Technology (NIST) Biometric Image Software 
(NBIS) is an open source biometric software that can be used to analyze scanned fingerprint 
images for image quality, minutiae detection, classifications, and generate fingerprint matching 
with a low rate of False Acceptance Rate (FAR) and False Rejection Rate (FRR). This software 
was developed for the Federal Bureau of Investigation (FBI) and Department of Homeland 
Security (DHS) (Ko & Salamon, 2010). Students used this open source software to match 
captured fingerprints and perform their statistical analysis. 

Packages containing a CanaKit Raspberry Pi 4 Model B with 4GB of RAM, 32GB 
MicroSD card, a 250GB USB drive, a 1.25” x 1.25” prism, and an autofocus camera attachment 
were supplied to each student. Detailed instructions were given on how to assemble and connect 
the Raspberry Pi to the autofocus camera. The Raspberry Pi OS (32-bit) version was 
downloaded and installed in the MicroSD card and the 250GB USB drive was used to store the 
NBIS libraries, programs, and database. 


~ 


Figure | - Assembled Raspberry Pi 4, prism, and autofocus camera 
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Figure 2 - Raspberry Pi optical fingerprint scanner 
Section 3.4 - Makerspace 


The Hatch It! Lab makerspace at the Center for Innovation and Entrepreneurship at the 
University of Tennessee Chattanooga served as an ideal space for prototyping the fingerprint 
reader. It houses six 3-D printers and the requisite design software for their operation. The 
makerspace is staffed by trained student workers, called Makerspace Managers, who 
troubleshoot, provide assistance in using the equipment, and police safety protocols. Students in 
this course received required training in the safe use of the makerspace. 

The makerspace is not merely a place to build a prototype. It is a safe space for 
experimentation. Students were able to fail in their efforts, reflect on what went wrong, and pivot 
to test a new approach. In this way, both the principles of the scientific method and the necessity 
of having an entrepreneurial mindset could be taught simultaneously. 


Figure 3 - Prototyping progression from cardboard to final 3D case 


Section 3.5 - Online Learning 


This course was taught during the spring of 2021 with an online synchronous modality 
due to the constraints imposed by COVID-19. Weekly class meetings were conducted over 
Zoom twice a week. Teams often met online to work on their design, presentation, and report. 
They occasionally met face-to-face on campus at the makerspace lab to print their 3D design and 
discuss improvements. 
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This online modality was challenging for the students and instructor. Difficulties 
included coordination, communication, and meeting to collaborate in the project. A major 
contributing factor to these challenges was credited to the online component and inability to have 
impromptu meetings before or after class because students were not physically in the same room. 


Section 3.6 - Pre and Post-Surveys 


In order to better understand the impact of our approach on learning of the students in this 
cybersecurity course, an anonymous pre- and post-survey was used to bookend this case study. 
The pre-survey sought to derive student perspectives on several topics prior to beginning the 
project. The post-survey had the same set of questions plus several additional sections with a 
slant on post experience and lessons learned. These additional sections were added to gather 
data on the student’s learning process and perspective on cybersecurity concepts, online learning 
experience, makerspace experience, and open-ended questions. This method permitted us to 
gather quantitative and qualitative data on topics such as student’s understanding of the learning 
objectives, teamwork, engagement, and demographics. A seven (7) point Likert scale was used 
to gather the quantitative data while specific questions and open-ended questions were designed 
to gather qualitative data. The class consisted of 35 students and 31 responded to the surveys. 
The pre-survey provided a baseline measurement and the post-survey data allowed for a delta 
analysis. 


Section 4 - Results 
Section 4.1 - Learning Objectives 


The surveys show that student’s confidence in their ability to meet the learning objectives 
to use the Raspberry Pi and the NBIS libraries to design a biometric system and 3D case to 
capture and analyze fingerprint images significantly increased by the end of the project. Most 
notable changes can be seen in the use of Raspberry Pi, code implementation, and use of NBIS 
libraries where it changed from 34% to 77%, 34% to 66%, and 29% to 74% in the agree/strongly 
agree ratings respectively. We also saw a shift in the 3D case design response from 11% to 43% 
in the strongly agree rating and can be attributed to increased confidence on themselves as shown 
by the changing numbers from pre and post-surveys in the somewhat, neither, and overall 
disagree ratings. 
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Use NBIS libaries for analysis Design 3D case 
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Figure 4 - Pre and post-survey Learning Objectives 
Section 4.2 - Teamwork 


In the pre-survey, students came into the project with a very strong positive perception of 
working in teams and their ability to do so effectively. Seventy-seven percent (77%) agreed 
(somewhat agree, agree or strongly agree) concerning the value of working in teams. The great 
majority of students saw themselves as potentially strong contributors to a team who would 
always come prepared and complete tasks on time (97% agreed). Students were almost equally 
confident in their ability to work well with teammates, understand the importance of team 
management (specifying roles and responsibilities), and understand the importance of 
communication in teamwork (94% agreed). 

In the post-survey, students largely perceived that they worked well with their teams 
(70% somewhat agreed, agreed, or strongly agreed), and that team members worked well 
together (74% agreed); however, in both cases, 20% or more disagreed. Students perceived that 
they were strong contributors to their teams (83% agreed) and that they completed tasks on time 
(90% agreed). By smaller majorities, respondents perceived that their attitude toward teamwork 
had improved (53% agreed) and that their team’s attitude had also improved (63% agreed); 
although, 23% disagreed that the team’s attitude had improved. Most students felt that they came 
prepared (90% agreed) and worked well with their team (83%). Sixty-four percent of 
respondents reported a positive feeling of accomplishment working with their team. 

While still positive, attitudes about teamwork in general, the ability to complete tasks on 
time, and being prepared all fell. It would seem that reality weakened perceptions of teamwork — 
its desirability and efficacy — to some extent. Overall, though, students were more positive than 
negative about teamwork at the end. Expectations for the power of teamwork can be inflated as 
individuals enter a project. Preparation by teams relative to establishing goals, roles, processes, 
and relationships may vary from team to team by the quality of the effort put into it. It is 
encouraging, however, that students concluded the project with a net positive attitude toward 
teams. 


Section 4.3 - Engagement and Learning Experience 


Student’s views that this was a good learning experience and motivation/interest dropped 
from 94% to 80% and 88% to 67% in the agree/strongly agree rating respectively. This change 
could be attributed to their experiences and challenges in teamwork as discussed in the previous 
section. Another contributing factor could be their technical preparation prior to this course. It 
changed from 18% to 39% in the overall disagree rating. Despite these changes in the negative, 
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84% feel motivate to continue learning about biometrics, 73% felt this was a good and effective 
learning experience, 66% recommend this project, and 76% felt the level of effort is adequate. 
An 87% of students believe this project improved their theoretical and applied understanding of 
cybersecurity and biometrics while 93% believe their critical thinking skills improved. 

Based on responses to the survey, students felt challenged by the project. For many, it 
was perceived to overtax their skillset. The anxiety produced by this challenge may have caused 
the drop off in the number of students who believed the project provided a good learning 
experience and was motivating for the rest of the course. Nevertheless, a significant majority of 
the class reported that they were more confident about meeting the course learning objectives 
after the project than they were before it began. Ultimately, two-thirds of responding students 
indicated that they would recommend a project like this one as a learning experience. 
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Figure 5 - Pre and post-survey Engagement and Learning Experience 
Section 4.4 - Cybersecurity Concepts (Post-Survey Only) 


Students were overwhelmingly positive in their responses associated with their 
understanding of biometrics and cybersecurity concepts after this project with 90% agreeing, 
87% agreeing they understand how theory connects to practice and implementation, and 
approximately 90% agreeing that they understand key biometric concepts of enrollment, 
verification, identification, false acceptance rate, false rejection rate, and hamming distance. 
These responses support this type of project-based experiential learning. 
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Figure 6 - Pre and post-survey Cybersecurity Concepts 


Section 4.5 - Course Online Experience (Post-Survey Only) 


connected with classmates, the instructor, and the course content while 77% agree that the 
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Engaging students in online classes has been a challenge throughout the COVID-19 
pandemic and 67% of respondents agree that online lectures through Zoom helped them stay 


project provided an opportunity to be an active participant in the online learning process. This 
online project-based experiential learning approach motivated 77% of the students to learn and 
research new cybersecurity topics and is in line with the reported 84% who felt motivate to learn 
about biometrics in section 4.3. When asked if they preferred this class be taught in a face-to- 
face modality, a surprising 40% neither agree or disagree while 47% agree. It could be argued 
that this was a good approach and delivery but further inquiries will help to clarify. 
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Figure 7 - Pre and post-survey Course Online Experience 
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Section 4.6 - Makerspace Experience (Post-Survey Only) 


After the project, 87% of the responding students felt they understood the tools of the 
makerspace. Their makerspace experience caused 87% to conclude that they were well-prepared 
to bring creative ideas to a project, while 89% felt they were able to use resources effectively and 
efficiently. Seventy-seven percent of students felt that they effectively managed their time and 
that they were adequately prepared to deal with failure. Students were largely positive about their 
ability to share what they learned with others (80% agreed). 

Over the course of the semester, it is clear from the survey that the students’ perception 
of their understanding of the makerspace tools increased substantially. Most came into the course 
not having been exposed to a makerspace. Requiring them to spend time in the space, working 
with the equipment, paid off in increased self-efficacy. Several students indicated that they 
would actively seek opportunities to use the makerspace again in the future. 

Interestingly, students’ confidence in their ability to manage time, to be prepared for 
failure, and to share their learning and offer assistance to others all fell off when they were 
confronted with the reality of actually using the makerspace. It is likely that these students were 
not prepared coming into the course for the numerous failed experiments and resulting pivots this 
kind of work entails. Given their own struggles, it is possible that reaching out to others became 
a luxury they came to feel they could not afford. These are all useful lessons that engaging in 
experimentation in a makerspace can teach. It is probable that a second makerspace experience 
would change this result, as expectations would be adjusted. 

Students’ perceptions about being prepared to be creative and to use resources to create a 
project improved somewhat. Because of the challenges presented by working in a makerspace, 
these students appear to have learned that they are, in fact, creative and resourceful. These are 
empowering life lessons that extend well beyond the walls of the makerspace. 
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Figure 8 - Pre and post-survey Makerspace Experience 
Section 4.7 - Open-ended Questions (Post-Survey Only) 


Several open-ended questions were given in the post-survey to give students a chance to 
freely express their opinions and collect additional qualitative data. When asked on their 
impressions of this project and its ability to teach Cybersecurity and Biometric concepts, most 
responses were extremely positive and indicated that the project provided an excellent learning 
experience, helped them learn the material, and challenge their skills. 
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The responses were mixed when asked for suggestions to help improve the project. 
Several focused on more instructions, more deadlines, making it an individual project, and 
implementing more accountability. These reflect the survey results described in sections 4.2 and 
4.3. Others responded that nothing was needed to improve on the project. 

Students were also asked if they envision using the makerspace lab again and 90% 
responded yes. The majority planned on using it for personal projects while some for both 
personal and class related projects. Those who responded no indicated that they had their own 
3D printer. The main tool and skill students learned from using the makerspace for this project 
was 3D modeling and printing. 


Section 5 - Discussion / Conclusion 


Our case study showed that despite the limited exposure to a makerspace prior to this 
project, students found it to be a useful learning tool and believe that this project provided them 
with a good experience and increased their confidence and understanding of cybersecurity and 
biometrics concepts. Students came into the project with some skepticism about teamwork, but 
the majority were positive about it post-project. 

We conclude that our research and use of the 5E Instructional Model and makerspace 
project-based experiential learning approach has shown benefits, and it is effective at bridging 
the gap and increasing students’ understanding of theoretical cybersecurity concepts and 
practical implementation. It is our intention to continue this study through several iterations of 
this course to build a dynamic database and allow for cross-course analysis. 
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Abstract: Teaching and learning have changed dramatically since the 2020 spring semester. 
Designing online courses and moving face-to-face courses online have been drawing attention in 
every educational institution. This article analyzes the challenges that we encountered when 
designing online courses at Purdue University followed by the introduction of solutions that we 
investigated. The final section of the article provides suggestions for future course design and 
studies. 
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Background 


Teaching and learning have changed dramatically all over the world since the 2020 spring 
semester. The pandemic created both challenges and opportunities in online teaching (Adedoyin 
& Soykan, 2020). Courses that were taught in classrooms had to be moved online or delivered 
through a hybrid method (a combination of online class and in-person class). This notable 
change created tension for both faculty and instructional designers. It has also produced 
opportunities to explore methods to make online instructions more effective, efficient, and 
engaging. This article will review the challenges that we faced from an instructional designer’s 
perspective and will discuss the methods, especially online teaching strategies, that have been 
investigated by the instructional designers in the course production team at Purdue University. 


Challenges and Opportunities 


In the past two years, the main challenges that faculty and instructional designers have 
encountered in our university can be concluded in three aspects. First, many faculty members 
did not have experience in teaching virtually, particularly fully online courses. There has been 
confusion on the concept of online teaching, and some faculty members were apprehensive 
about converting their traditional in-person classes to online class. Second, a wide range of 
instructors were not familiar with using learning management system (LMS). This situation 
became extremely complicated and increased tensions as our university started to move to a 
new LMS in January 2020. Finally, faculty members have expressed concerns about assessing 
online learning. One of the biggest concerns was how to maintain academic integrity in ways 
that are comparable to an in-person class assessment. 


While working with faculty, instructional designers realized the second and the third challenges 
can be solved by providing workshops, consultations, and related technical support. However, it 
takes longer time and more efforts to overcome the first challenge, which is the lack of online 
teaching experience for many faculty members. Additionally, researchers pointed out most 
university faculty lack formal trainings compared to teachers and have been using more 
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unofficial methods that they have been taught, such as in many STEM courses (Yang, 2017). 
They have been caught in more difficult situations while attempting to design online courses. 
Given this situation, designers worked together with faculty by analyzing online teaching 
phases. Similar to the methods of teaching traditional in person courses, online teaching 
includes three phases, which are planning instruction, delivery of instruction, and evaluation of 
student learning. Greater emphasis should be placed on the initial planning phase (Simonson, 
Smaldino, Albright, & Zvacek, 2012) due to the teaching strategies and evaluation plans needed 
to complete this phase. Instructional strategies refer to the methods instructors use to help 
students achieve learning goals (Smith & Ragan, 2005), and they are considered as critical 
factors that impact online teaching and learning (Fresen, 2005). Therefore, investigating online 
teaching strategies became one of the first steps in our course design process. 


Investigating Online Instructional Strategies 


Educators have discussed online instructional strategies and provided suggestions, such as 
encouraging student interactions (Miller, 2007), using student-led discussions (Simonson, 
Smaldino, Albright, & Zvacek, 2012), providing prompt feedback (Sorensen & Baylen, 2004), 
and adapting some methods that were proven to be successful in a traditional classroom setting 
(Simonson, et al, 2012). These discussions and suggestions inspired our investigation. While 
working with faculty, our designers adopted the backward design method (Richard, 2013) and 
followed three principles to choose/create instructional strategies, which are connecting the 
course activities, mapping the course content, and engaging class online. Each of these principles 
includes multiple strategies. 


Connecting Course Activities: 


The most popular strategies that we have been using to deliver the content in fully online 
courses include using online discussion boards for class discussion, creating course Q & A 
(online discussion board) to encourage students asking question, recording lecture videos to 
guide learning, and providing synchronous session/virtual office hours. We noticed that these 
activities are usually designed independently to each other and often lack in connections 
between each other. For example, some faculty members complained that some students were 
working on the homework without watching lecture videos. One way we have solved this 
problem was to add in-video quizzes or after-video quizzes. Another way we tried was by 
connecting the lecture videos with online discussions. In other words, instructors could ask 
questions at the end of the video and students would be required to post their response on 
discussion board. The faculty noticed that the number of views on lecture videos have increased 
significantly after adopting this strategy and concluded this was a better way to assess learning 
progress. 


In fully online courses, office hours also need to be held virtually. However, faculty members 
have reported that this is not as efficient as regular face-to-face office hours. Students seemed to 
be unwilling to ask questions using a video conference tool. The solution that we found was to 
connect the Course Q & A online discussion forum directly to virtual office hours. Students can 
choose to post their questions on the forum before office hours. Faculty review the questions 
and answer the ones that are easy to explain in the course Q & A forum and lead a deeper 
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discussion on the more complex questions in the video conference. This method has reported to 
be extremely successful and resulted in a higher efficiency of virtual office hour usage. 
Therefore we kept working in this direction and added small element to the live (synchronous) 
classes. We encouraged instructors to post the topics and detailed plans of the live sessions on 
the course site to help online students better prepare for the live class and discussions. The 
following images show a synchronous session plan and a virtual office hour plan that have been 
posted on two different course sites. 


D 
Synchronous Sessions Plan: 


¢ Zoom meeting at 5:30 p.m. Thursdays. 
Meeting topic 1: Introduction to the OECD and IMF Interactive Tools 
Meeting topic 2: How should the U.S. design a system for risk spreading and compensation 
for pandemics, and what roles should insurance and government play in that system? 
Meeting topic 3: Introduction to the course Q&A 

¢ All the meeting will be recorded and posted under the Zoom Sessions tab. Please check the recordings 


if you could not attend the meeting. 


Figure 1: Synchronous Session Plan. 


My virtual office hour will be on Wednesday, 4:00-5:00 p.m.. Please feel free to meet with me 
through Zoom if you have any questions. Click here to join the meeting. Or use Meeting ID 


The topics of Discussion: Discuss any questions regarding the virtual lab environment setup; 
Describe the chosen prefix attacks and birthday attacks on hash functions; Exemplify the need for 


security against CPA and CCA attacks in the networking situations. 


A (Zoom) video recording of this synchronous meeting will be uploaded under Virtual Office Hour 
Recordings tab. 


Figure 2: Virtual Office Hour Plan. 


Connecting course activities can also engage students in hybrid course settings. The instructors 
of a hybrid MBA course designed a group assignment for students to draft discussion questions 
based on the readings and learning resources in the first week of class. This assignment was 
required to be submitted before the end of the week. In the following week, students met in 
classrooms and each small group led the class discussion using the questions that they drafted in 
the assignment. This was reported as a very effective way to engage hybrid students and we are 
planning to adapt this strategy for future fully online courses. 


Mapping Learning Objectives 


Instructional design should follow a well-organized procedure that provides guidance to 
instructors and students (Simonson, Smaldino, Albright, & Zvacek, 2012). For instructors, one 
way to organize the course materials is to create clear course outcomes and unit/module level 
learning objectives. Learning objectives are described as statements that explain what students 
should be able to do after they have completed a segment of instruction (Smith & Ragan, 2005). 
Well-written learning objectives are specific, measurable, and can provide a clear road map of 
the course content and assessment. 
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Learning objectives not only help faculty organize course content but also helps to improve 
learning. In traditional in-person class, stating learning objectives in class is considered to be an 
effective teaching strategy (Englert, 1984). Research shows explicitly linking learning 
objectives and class activities helps motivate and engage students (Reed, 2021). In addition, 
sharing learning objectives with students in each lecture and aligning objectives with 
assessments helps increase student academic performance (Englert, 1984). Research studies 
shows that students consider the list of learning objectives as the most helpful component of a 
lecture (Armbruster, Patel, Johnson, & Weiss, 2009). 


In both online and in-person course design, we tried to map course content starting with helping 
faculty create two levels of learning objectives: the course level learning outcomes and the 
weekly module level learning objectives. For example, in a computer science course, course 
level objectives are listed under course introduction page and the module level objectives are 
listed under each lecture video and assignment. The following image shows the connections 
between module level learning objectives and lecture videos. 


Video Lecture: Government Spending Around the World 
Run Time: 12:59 


PowerPoint: Lecture 1.1 


Objectives: after watching this video, students will be able to 


¢ Summarize the trend in levels of government spending and tax revenue over time and how 


the U.S. compares to other countries (LO1.1) 


Figure 3: Learning Objectives 


Lecture 2.3 Block Ciphers (19:58) 


v 
@& External Learning Tool 


in 
(=) 
=p 

ais Read the following Learning Objectives before you start to watch the lecture video (click on the above 
"Lecture 2.3_ Block Ciphers" link to watch the video): 


At the end of this lecture, students will be able to 


© Recognize the need for block ciphers for secure communication 


© State prominent block cipher modes and their comparison 
These objectives are directly connected to the Course Outcomes 3. 


© CO3: Identify the appropriate defense mechanism(s) and its (their) limitations, given a network threat . 


Figure 4: Learning Objectives and Course Outcomes 


Adapting In-person Class Activities for Online Courses. 


As mentioned earlier, many strategies that have been successfully used for traditional in-person 
classes can be adapted for online courses (Simonson, Smaldino, Albright, & Zvacek, 2012). 
Research shows that many methods can help improve learning efficiency in traditional in person 
classes such as group work (Chad, 2012), case-based class discussion (Mackavey, & Cron, 
2019), project-based learning (Langer-Osuna, 2015), and flipped classroom (Zheng, Bhagat, 
Zhen, & Zhang, 2020), etc. In addition, the Seven Principles (Chickering & Gamson, 1987) of 
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good practice in undergraduate education have been proven to be successful in traditional in- 
person classes. These principles are: 
e 1. Encourage contact between students and faculty 


2. Develop reciprocity and cooperation among students 

3. Use active learning techniques 

4. Give prompt feedback 

5. Emphasize time on task 

6. Communicate high expectations 

7. Respect diverse talents and ways of learning (Chickering & Gamson, 1987, pp.3) 


In the past 20 years, these principles have been adjusted to fit online education (Sorensen & 
Baylen, 2004). The strategies that focus on student center learning and incorporate 
communication and interactions are considered the most successful methods (Miller, 2007; 
Simonson, Smaldino, Albright, & Zvacek, 2012). While designing online course instructional 
strategies, we adapted the seven principles for online courses and focused on encouraging 
communication. The following table shows the specific strategies that we have been using in 
online course design. 


Table 1 
Online Course Instructional Strategies 
eeve z Adapted Instructional Strategies for Online Courses 
Principles 


Student -faculty 
contact 


e Set up Course Q & A forums under discussion boards. In many 
courses, the faculty only chose to set up Q & A forum for each 
module. 

e Provide weekly virtual office hours using video conference tools 

e Provide synchronous class sessions using video conference tools 


Collaboration 
among students 


e Set up self-introduction forum and random topics forums under 
the discussion board 

e Assign online group discussion questions and encourage online 
communication 

e Set up group projects and homework 

e Use peer evaluations to encourage participation 


Active learning 


It refers to giving students opportunities to think about a topic and 
respond to learning content (Sorensen & Bylen, 2004), such as case study 
analysis and structured discussions (Simonson, Smaldino, Albright, 
Zvacek, 2012). 

e Create virtual presentation homework 

e Utilize peer evaluation and critique 

e Create simulations or case study projects 

e Connect online discussions with live classes to improve deeper 

understanding 


Prompt 
feedback 


e List feedback turnaround time on syllabus and course site 
e Host virtual office hours to answer questions 
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Time on task e Clearly list due dates on course site 
e Use checklists to help students manage their time 
e Use weekly task lists to remind current learning tasks 


High e List extra reading and learning materials for students who would 
expectations like to explore more relating to specific topics. 
e Encourage students to explore more if they are interested in any 
topics that are covered by the learning materials. 


Respect for e Create self-check quizzes and review documents/resources to help 
diverse talents students review the content that have been previously 

and ways of covered. (These activities won’t count for the course final grades). 
learning e Provide options on the assignment topics for students to choose, 


especially on the final project (For example, the final presentation 
can be pre-recorded, present in a synchronous session, or other 
options such as a term paper) 

e Record virtual office hour meetings and live sessions and upload 
them to the course site for students who did not attend 

e Provide a wide variety of instructional strategies to meet students’ 
needs. 


Conclusion 


Working together to solve the challenges that we encountered in online course design helps 
faculty realize that the key concepts to successful online teaching and learning are in the design, 
development, and delivery of instruction (Dempsey, & Van Eck, 2007; Kidd, 2005; & Simons, 
Smaldino, Albrigh, Zvacek, 2012). In addition, with a high-quality instructional design, online 
learning can be just as or more effective compared to traditional learning (Colvin, Champaign, 
Liu, Zhou, Feredricks, & Pritchard, 2014). The three principles and the related strategies that 
are discussed in this article focus on engaging online student and encouraging communication, 
which is considered as one of the most important principles for online teaching (Sorensen & 
Baylen, 2004). However, the effectiveness of these strategies is greatly impacted by many other 
factors, such as the course content, the way faculty diver the course, and student motivation. We 
would like to see more studies on the effectiveness of these specific strategies in the future. 
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Abstract 


The importance of precision medicine is increasingly being recognized in healthcare. 
Precision medicine is driven by patient data and physicians’ diagnoses in a comprehensive 
manner that considers the uniqueness of individual patient where each patient is a partner in 
the whole process. Big data analytics can provide a means to analyze and interpret healthcare 
data in a manner that can be quickly implemented in patient care because the available data 
is not structured in the way traditional databases are. There is growing interest by physicians 
to take advantage of big data analytics. However, limitations in deciphering and interpreting 
this data by healthcare professionals has impeded implementation of this technology. Most 
off-the-shelf software do not provide step-by-step instructions needed for a physician to 
understand big data analytics. In this paper, we provide a way to create big data 
visualizations through MongoDB with upload & download capability on web repositories. 
Keywords from PubMed were integrated to provide data visualization using the MongoDB 
programming thereby providing a unique solution for the issues that healthcare providers 
face in their understanding of the big data. The web repositories with big data visualizations 
for precision medicine will provide healthcare professionals and specialists a readily 
accessible platform for efficient diagnosis and care. Recommendations are provided about 
patient documents and visualizations, which will provide a thorough understanding of the 
data, knowledge sharing, collaboration, help in medical education and efficiency to 
healthcare providers. 


Keywords: big data visualization, mongodb, precision medicine, web repositories 
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Introduction 


The history of precision medicine dates back to 1960 (Jane, 2002). However, the term 
“personalized medicine” first appeared in published works in 1999 (Managed Care, 2011). This 
term was revived through the Personalized Medicine Initiative (PMI) by President Obama in 2015 
during his State of the Union address. The traditional one-size-fits-all approach with patients’ 
diagnoses needs to be improved, as it lacks the inclusion of the latest technologies and 
collaboration. Precision medicine provides a newer approach to patients’ diagnoses that is 
affordable and provides an opportunity to consider other important factors, such as genes, 
environment, and ethnicity which could provide useful information for the treatment (Dhawan, 
2016). One reason why precision medicine is uncommon in routine practice is because analyzing 
big data is complex and beyond the scope of most physicians. In a 2020 Health Trends Report 
done at Stanford University with 523 physicians, 44% of them said their medical education was 
“not very helpful” or “not helpful at all as it applies to the emerging data related technologies 
(Stanford Medicine, 2020). This response came in the context of the latest data driven technologies 
for healthcare. The same report pointed out the fact that the majority of the physicians were open 
to the latest technologies dealing with data and thought it was essential for their decisions on 
healthcare. However, the challenge is in the understanding and use of the latest technologies that 
could be used in healthcare 

There is enough evidence about the need and the importance of precision medicine and big 
data analytics (Jain, 2002 & Stanford Medicine, 2020). The problem lies when the physicians do 
not have sufficient skill sets to understand big data and its analysis as it is beyond the reach of 
most physicians. While a good percentage of physicians are going back to school for further 
training on the latest technologies, it might not be possible for every physician to do the same. 
Big data analysis through an easy-to-understand software might be a solution for those who do 
not have time to go back to school. This paper presents the methodology for the design of such a 
software through big data analysis using MongoDB visualizations. It is critical that the 
physicians who would utilize such a software-are able to understand the patients’ data and create 
visualizations without learning all the technicalities of MongoDB. A concept of web repositories 
is also presented in this paper, demonstrating that visualizations could be stored for collaboration 
among other physicians, healthcare professionals, and big data experts — we call them a “team” 
of precision medicine in this paper. Such a “team” can benefit enormously through an easy-to- 
understand software which is designed for an average layman person and not for IT experts. 


Methodology 


The main topics of this paper include patients’ documents processing, extraction of the 
data from patients’ documents into JavaScript Object Notation (JSON) format, the storage of 
JSON into big data using MongoDB, the creation of visualizations using Python programming 
language, and the upload and download capabilities of the proposed software to an intranet web 
repository using Python. While JSON, MongoDB, Visualization, and Python are done by an IT 
expert — it does not have to be learnt by a physician. The physician is an end user of the software 
that used these technologies. These technologies are currently used by big data on daily basis 
(Agraharil & Rao, 2017). There are so many technologies which are out there to process medical 
data, however, it needs a bridge building between the IT experts and the medical community. 
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Only then, the potential of these emerging technologies could be utilized benefitting the 
humanity. A software approach proposed in this research is the solution. 


The process of the proposed software presented in this paper starts by finding patients who 
would be interested in participating in a collaboration program similar to All-of-Us (NIH 
Homepage, 2020). The patients sign a consent form and provide the relevant information needed 
by physicians on the team. It is suggested that a big data analyst/s, such as a MongoDB expert is 
hired by a healthcare organization’s team depending on its size. In May 2018, the A//-of-Us 
Research Program was declared and funds were allocated to collect the whole-genome 
sequencing of 200,000 people per year (NIH Homepage, 2020) . After the patients sign consent 
form to become a partner in a program like All-of Us, the patient data is collected by his or her 
healthcare provider and stored in documents of history and physical exams, operative notes, 
discharge summaries and outpatient clinic visit notes (NIH Homepage, 2015). This document is 
then provided to a MongoDB expert. The MongoDB expert extracts the pertinent big data in 
consultation with the physicians who are on the team of precision medicine. MongoDB uses a 
JSON type description, which is based on key-value pairs. A sample big data obtained from a 
patient’s document could be like figure 1 (NIH Homepage, 2015). A similar approach in storing 
big data as JSON format into MongoDB has been proposed previously by other authors as well 
(Messaoudi, Fissoune, & Badir, 2018). JSON is widely used in the IT industry for many text 
related data. In the past, XML was a choice for small to medium size data but JSON has more 
usage in the modern IT software. 


{ 
Id : 123456789 
Age: 59 
Gender: M 
Ethnicity: Black 
Marital Status: Married 
Location: Denver 
Physical Activity: Yes 
Pain Scale: 5 


} 


Fig. 1. A sample JSON format for the data extracted from patients’ documents and stored in 
MongoDB 


JSON is a popular format used in web and database related software technologies. 
MongoDB is a NoSQL (Not-only SQL) database management system used for documents. 
Traditional database management systems lack the capabilities to handle documents’ attributes 
(Messaoudi, Fissoune, & Badir, 2018). Traditional databases involve relationships between 
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entities (tables). Traditional databases had been with the IT people for quite some time. While it 
will stay in many disciplines, the production of big data on daily basis demand for a new 
technology like big data. The above information was chosen as if a physician wanted to pick data 
related to Pain_Scale based on demographics. The Pain_Scale was fictitiously entered from a 
scale of | through 10 which was multiplied by 10 to create better visualization. Visualization is 
achieved here through python modules. Python is another popular language used in the 
development of software which has built in modules to help/display visualization as shown in 
figure 2. Visualizations are easy to understand by everyone including physicians. They do not 
require the technicalities involved behind the scenes. Visualizations are available in various 
forms of graphs, such as, line, bar, histograms, etc. Many visualizations were used in the current 
COVID-19 data using python modules (Ganesh, 2021). These visualizations provided the 
healthcare experts to see the trends of the pandemic. Similar visualizations can be used in the 
proposed software to understand the patients’ data under examination. These visualizations are 
then shared by the ‘team’ to get the best expertise in the area. Many case studies can be looked 
at by various physicians for knowledge sharing and best diagnosis. It is a game changer concept 
in the healthcare industry. Right now, most of the case studies are done by mutual acquaintances 
among the physicians. However, with the software proposed by the author these visualizations 
can be shared through hundreds and thousands of physicians across the globe using the concept 
behind web repositories. With the technologies mentioned in this paper, all of this is possible and 
feasible. 


— ju pyte r Untitled73 Last Checkpoint: 3 minutes ago (unsaved changes) 


File Edit View Insert Cell Kernel Widgets Help 
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In [3]: import pandas as pd 
Pain_Scale = pd.read_csv(“c:\Python_Practitioner\Day6\Pain_Example.csv™) 
Pain_Scale.plot() 


Out[3]: <matplotlib.axes. subplots.AxesSubplot at @x24a58615048> 


160 — Black 
—— White 
=—— Pais 


Fig. 2. A plot showing visualization of patient data. The MongoDB key-value pairs are converted 
into csv format and then python programming is used to draw the visualization using Jupyter. 


In the above sample data, 8 key-value pairs were used. However, this data could be 
structured involving more key-value pairs as needed. The JSON file that stored the key-value 
pairs of data was then converted to a comma separated value(csv) file by using Python 
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programming language. The Python programming code used is shown in figure 2 to create the 
visualization from the data originally stored in MongoDB. 


The pain_scale is between 9 through 58 which were rounded to 10 and 60 respectively. 
The different frequency for Black and White were entered for a certain pain_scale. All of these 
numbers were fictitious to demonstrate that a visualization is possible for big data (MongoDB) 
using Python. The Python code in Figure 2 imports “pandas” which is open source, meaning it is 
free of charge to its users. Jupyter is used as an Integrated Development Environment(IDE) to 
run the Python code. Python has rich libraries and modules like pandas, which have built-in 
methods to help plot the visualization as well. A physician, after consultation and training on big 
data should be able to create a similar visualization about “Pain Scale” as it relates to the 
demographic data. MongoDB charts and many third-party software tools also allow MongoDB 
data to be used for visualizations. The visualizations illustrate the level of Pain Scale among two 
key-value pairs based on their ethnicity. These two pairs of data are a sample for demonstration 
purposes and can be changed to other key-value pairs on a as needed basis. The whole process 
from the collection of patients’ data to the creation of visualization is shown in Figure 3. 


Patients' Data 
Physicians 
+ 
t | Healthcare Providers 
Documents . a 
J Consultation/Training t t 
Big Create Verify 


MongoDB -_ ——> MongoDB Experts 


Data 
Se « } 4 
Verify : = 
Ny, Visualizations 


Fig. 3. Schematic outlining the whole process from the collection of big data from patients’ 
documents until the creation of visualization is shown. 


In Figure 3, the MongoDB experts receive the patients’ documents in electronic format 
and convert into JSON format. The JSON format is ready to be stored as big data into 
MongoDB. During this process, the MongoDB expert is in constant consultation with the 
physicians, providing consultation and training on what data is being stored that is relevant and 
important. Once trained, the physicians can create visualizations themselves that are ready to be 
uploaded. During this process, the physicians verify the accuracy of visualizations among their 
peers and MongoDB experts. 


The process of collaboration among physicians begins when a physician starts uploading 
and downloading such visualizations and shares with other physicians and healthcare 
professionals. In the proposed software, a physician would have to provide a password for 
authentication before he or she can upload or download a visualization on a web repository. Web 
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repositories provide a great way for sharing knowledge and collaboration in many areas 
(Siddiqui, 2015). The visualizations will be stored on an intranet web repository based on its 
classification. Classification plays an important role in searching and saves time (NIH 
Homepage, 2015). The web repository is a central location like a Microsoft SharePoint intranet 
site (Williams, 2011). The classification could correspond to folders. A sample web repository 
would look like figure 4 where a physician will click on the visualization of his/her choice after 
logging in. 


, ——— 
Pain Diabetes Cancer 
| Visualization Visualization 


Migraine +****= Stomach 


Fig. 4. The web repository classified into different categories of diseases. 


Once a visualization is completed by a physician, they can authenticate their identity 
though a username and password for the intranet that they are part of. This is essential for the 
security of the data of their patients. The visualizations are classified based on the type of a 
disease. The classification is critical for an efficient use of information. With the overwhelming 
amount of data that is available, it is difficult to filter the data that is relevant. Therefore, the 
diseases are put into their corresponding folder shown in Figure 4. This also gives an opportunity 
for the physicians to click on the folder of their specialty. This classification is done by the 
MongoDB experts after consultation with the team. It is possible to have further classification 
within a disease category as well. 


The whole process after the creation of a visualization to uploading/downloading on a 
web repository is shown in Figure 5. The visualizations will help physicians to understand the 
disease with more data and help in the diagnosis of a patient in a personalized manner. 


= Download/ eaters ee eaeet hi 
Physicians National & International 
+ “== Upload) ==> Web Repositories 
Healthcare Professionals _ Collaboration 


tft 7 


Download/ 
Authentication Upload/ 
Collaboration 


Web Repository 
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Fig. 5. After the visualization is created, a physician is authenticated through username/password 
and then given the option to upload or download on an intranet for collaboration with other 
physicians. This concept could be extended to the national and international level web repositories. 


As shown in Figure 5, once a visualization is uploaded, it can be shared with other 
physicians and healthcare providers. The power of collaboration through web repositories 
should be limited to the local doctors’ offices and healthcare providers. It should be expanded to 
the national and international level web repositories in situations where the cure is not easily 
found as in the case of the COVID-19 outbreak. 


Results and Discussion 


While all of the data related technologies discussed above already exist, and many works 
have been published on each of the above topics, it lacks the comprehensive approach that is 
needed to benefit from each of the above technologies at smaller doctors’ offices and healthcare 
organizations. Moreover, the element of collaboration is also lacking at smaller medical offices 
in the whole diagnostic process of a patient. The objective of this paper is to synthesize all the 
emerging technologies into one easy-to-use software that will benefit the physicians who would 
otherwise not pursue a data emerging degree program and would be excluded from the power of 
collaboration and modern technologies. The key thing is to develop a sophisticated software 
using the emerging technologies. 


Precision medicine is becoming the future of treatment for the patients in developed 
countries where the latest data related technologies have progressed tremendously in the past 
decade. Many physicians are realizing the power of big data and its application in the field of 
precision medicine. While a good percentage of the physicians are going back to school for data 
technologies related degrees, not all of them have this flexibility. A user-friendly software with 
little training for physicians is proposed in this paper to better equip physicians who are unable to 
pursue another big data technology degree. The software proposed could be written with the help 
of IT experts, physicians, and healthcare professionals who are part of the same team. The 
technologies involved in the writing of the software include, python programming with 
visualizations and big data using MongoDB. These technologies are proven in the industry and 
are widely used for the software that deal with big data. 


Patients should be notified requesting their consent to be included into a research study 
about a certain disease before their documents are accessed for the proposed software. The key- 
value pairs in the MongoDB are selected from the patients’ documents, which are those fields 
that are useful to physicians’ diagnoses process. This paper picked one variable of pain-scale 
randomly, and fictitious responses from two ethnicities were stored in a csv format. Python was 
used to visualize the csv file. Python scripts can be used to create a button in a software’s user 
interface for upload/download capabilities that could be merged with the proposed software 
(More complex APIs, 2020). The physicians are authenticated with a username/password before 
they can access the web repository in an intranet. The participating physicians could collaborate 
in a similar fashion through web repositories on the internet at national and international level for 
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a particular disease. Collaboration is an important factor to learn about a disease and diagnoses 
(NIH homepage, 2020). The collaboration will help physicians to understand the disease with 
evidence from more data, help gain expertise from their colleagues, and allow faster and more 
accurate diagnoses of their patients’ illnesses. All of the technologies discussed in this paper are 
available and tested by the industry. The thing which is missing is the connection of these 
technologies with the physicians. There is no better way to communicate and collaborate than 
web repositories in the modern age when we are all connected through today’s cutting-edge 
internet technology. 
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Abstract 


Learning analytics provide valuable information for learners and instructors by 
combining and analyzing learners' historical data during the learning experience. The 
most common way of employing this information is in the form of learning analytics 
dashboards (LADs). This study primarily aims to propose LADs design based on the 
perspectives of various stakeholders. The secondary aim of the study is to propose the 
concept of ‘learning analytics feedforward’. After an iterative and formative design 
process, the LADs were developed in two different interfaces: a course-related 
dashboard and a topic-related dashboard. Each dashboard element is classified 
according to whether it contains feedback or feedforward. The development of LADs 
based on learner expectations and lecturer perspectives is described in detail. 


Keywords: learning analytics, feedback, feedforward. 
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1. INTRODUCTION 


With the spread of technology, the interest in online learning environments is increasing. The 
desire to obtain meaningful results from the digital data left by learners in online environments 
and the efforts to improve learning environments reveal the need for learning analytics. 
Learning analytics is based on data resulting from the user's interaction with information and 
communication technologies. For example, recorded log data is potential data for event 
learning analytics with timestamps about viewing certain resources, completing essays and 
quizzes, or discussion messages viewed or sent (GaSevic et al., 2016). Learning analytics is 
considered an interdisciplinary field within the fields of educational technology, pedagogy, 
machine learning, business intelligence, artificial intelligence, and statistics as a new field of 
study (Guenaga & Garaizar, 2016; Siemens, 2013, Chatti et al., 2012). The aim of learning 
analytics is to improve learning, teaching, and learning environment by using educational data 
(Clow, 2013). Moreover, learning analytics is expressed as measuring, collecting, analyzing, 
and reporting data about students and their contexts to optimize learning and learning 
environments (Siemens, 2013). 


The use of learning analytics has been increasing in recent years and is frequently preferred 
especially for the creation of individualized learning environments. In this process, learning 
analytics indicators are important in terms of monitoring students' success-failure situations 
and monitoring their behaviors in the process. In addition, recommendation and guidance 
feedback based on learning analytics (also known as learning analytics feedback) by instructors 
in this process will provide various benefits to students. The most foreseen benefit of the 
learning environment is the improvement of the communication between the instructor and the 
student. In addition, the instructor can give more effective feedback based on the knowledge 
gained in the process. The log data consists of the students' own behaviors, and feedback can 
be provided on the behavior of the students, such as which lesson and when they watch, where 
they hang out, where they do it right. Siemens (2013) states that learning analytics will affect 
existing education models and provide new insights on learning and teaching. He further 
indicated that in order to accomplish this goal, firstly it is necessary to make a deep sense of 
the existing potential in education, and secondly to deal with the difficulties encountered in 
educational applications of learning analytics. 


Instructors can obtain information about students' behavior, performance, learning processes 
and learning outcomes in the online learning environment which can be obtained by learning 
indicators. In addition, these reports can provide instructors with an insight into students' 
learning needs and learning deficiencies. Also, instructors can have a foresight about whether 
to intervene with the student based on the learning analytics results in the process. Furthermore, 
how and when this intervention will take place is decided based on these results. To do so, 
creating personalized learning environments with learning analytics can be utilized as useful 
tools to provide personalized feedback. In this context, learning analytics can also be 
administered as an evaluation tool regarding the instructional design process, and it can provide 
input for the next cyclical processes of the instructional design process. 


Recent research results yielded that providing tips, advice, and guidance about learning 
behaviors by using learning analytics in the process of creating personalized learning 
environments and improving instructional design processes is recommended (Jivet, Scheffel, 
Specht, & Drachsler, 2018). Thanks to the learning analytics based recommendation and 
guidance feedback, students will be able to recognize their own learning deficiencies and will 
try to tackle them by knowing where they have shortcomings and mistakes. The 
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recommendation and guidance feedback utilized are based on learning analytics. These 
feedbacks given in the process are suggestions based on learning analytics. 


Feedback is considered the key element in formative assessment (Carrillo-de-le-Pena et al., 
2009). However, feedback alone is not sufficient for formative assessment. Similar to 
instruction, formative assessment is often sequential (Hattie & Timperley, 2007). Each new 
sequence exponentially increases the possibilities for the next. Therefore, a feedforward 
approach is needed in addition to feedback. 


1.1. Learning analytics feedforward 


The concept of feedforward is as old as feedback. Bjérkman (1972) defined feedback and 
feedforward as different operators serving the same purpose. In both operators, task-related 
information is provided by evaluating current performance against a specific target. Both can 
have supporting functions in providing information and policy making in the teaching process. 
(Sengupta & Abdel-Hamid., 1993). It is a controversial issue to completely separate the 
concepts of feedback and feedforward. While some views on the meaning of feedforward 
consider this concept as responses to feedback or as a feature of effective feedback (Dulama & 
Tlovan, 2016; Faulconer, Griffith, & Frank, 2019; Hattie & Timperley, 2007), some view a 
computational strategy used to offer suggestions before a task or performance (Bjérkman, 
1972; Hendry, White, & Herbert, 2016). 


Feedforward was not investigated as commonly as feedback in educational research (Dulama 
& Ilovan, 2016). The lack of necessary environments for providing feedforward was one of the 
reasons why it was not widely used in the past. However, in recent years, feedforward has 
gained attention by the use of educational data mining algorithms based on machine learning 
in learning environments (Knight, 2020; Meredith, 2020; Sedrakyan, Malmberg, Verbert, 
Jarvela, & Kirschner, 2020). 


When it comes to operational definitions of feedback, the focus is always on current 
performance or target performance. However, there is a need for proactive approaches in the 
context of formative assessment. It is necessary not only to focus on current or target 
performance, but also to take into account possible performance. If possible performance is 
predicted accurately, prevention of learners from drop-out will be more easier and intervention 
to learning experience will be more meaningful. 


The study primarily aims to propose a learning analytics dashboard design based on the 
perspectives of various stakeholders. The secondary aim of the study is to propose the concept 
of learning analytics feedforward (LA feedforward) to the educational researchers. As 
mentioned before, although it is not frequently used in educational research, the concept of 
feedforward has different functional definitions. For this reason, we recommend the use of the 
concept of learning analytics feedforward, just like the concept of learning analytics feedback 
(LA feedback). Thus, a common view will be formed when LA based feedforward is said. 


2. METHOD 


Within the scope of this study, a systematic data collection process was carried out for the 
dashboard design. This process was carried out on the basis of design-based research. Design- 
based research; new theory with iterative processes that do not have a fixed prescription (Barab, 
2014, pp. 151), requires high cohesion and cooperation with the participants (Amiel & Reeves, 
2008), is aimed at improving educational practices (Wang & Hannafin, 2005), and can be 
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adapted to other teaching contexts. It is a research method/framework that aims to develop 
applications (Barab, 2014, pp. 151). In this study, analysis, design and development phases 
were carried out. Although it is an iterative and formative design process, this study can be 
defined as quasi design-based research since there is no implementation phase. 


2.1. Participants 


The study was carried out in three phases with different participants. Figure | shows 
the distribution of participants at each phase by gender and education level. 


Male : 2 
Female: 5 


Male : 8 Male: 3 
Female: 14 Female: 3 


All of them 
graduate students 


Undergraduate: 9 Graduate: 3 
Graduate: 15 PhD: 3 


First Phase Second Phase Third Phase 
Learner Lecturers' Design Learners' Design 
Expectations Evaluation Evaluation 


Fig 1. Phases of the study and participants 


2.2. Data collection 


In the first phase of study, an inclusive question was used to determine learner expectations 
from learning analytics. The expectations of 22 undergraduate and graduate students were 
obtained via a web-based form. This form includes only the below question: 


“We need you to use your imagination. Suppose there is a Genie in the system, like Alaadin's 
Genie. If you had a maximum of 3 wishes (information you want to see based on your system 
interaction data), what would you wish from this Genie?” 


A draft design for dashboard elements was created based on these expectations. The lecturers’ 
perspectives on the draft design were gathered in the second phase of the study. Draft design 
form was sent via e-mail and six of eight lecturers participated. Draft design form includes this 
information: 


Purpose and scope of feedback/feedforward to be presented, 
Metrics/variables needed to present feedback/feedforward, 

Details of the metrics/variables calculations or visualization components, 
Appropriate data visualization graphs/charts to present feedback/feedforward, 
Area for the lecturer comments, 

Lecturers rating area for evaluating the information chunk from | to 5. 


DO ge 


Lecturers gave their opinions for each dashboard element containing the above information, 
which was created according to learner expectations. Additionally, the lecturers can suggest a 
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new dashboard element in the above format. This form was revised as a result of the opinions 
of the lecturers. 


The revised draft design form was presented to the students in the third phase of the study. The 
dashboard elements in the draft design were rated by seven graduate students who also took 
part in the first phase. The students on the other hand were not provided with metrics, detailed 
explanations and a comment area. Students only rated it from | to 5. Elements rated as 4 or 5 
by all students were the elements to be included in the dashboard to be developed. 


2.3. Environment 


The aim of this study is to design a dashboard to be integrated into an existing MOOC system. 
However, in order to better understand some of the metrics and system components in the 
dashboard, Smart MOOC Integrated with Intelligent Tutoring (SMIT) is briefly introduced in 
this section. 


SMIT was developed using HTML, JS, CSS, PHP, MySQL script and software languages. On 
the SMIT platform, the topics are presented as modules. Learners must take a mastery test to 
complete each module and the system decides that the relevant learner is master for that topic. 
Bayesian network method is utilized to determine the level of mastery. SPRT (Sequential 
Probability Ratio Test) was utilized in the estimation of mastery. When SMIT makes an 
authorized decision about the user, this is indicated to the user. When the system decides that 
the learner is not the master, the learner is directed to the relevant content and recommended 
to study the content. The user who is directed to the content decides that he/she is ready by 
browsing the learning materials, he/she can take the proficiency test until he/she is the master. 
When learners are not the master, they can also be directed to the intelligent tutoring system. 
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Fig 2. Screenshot of the learning environment 

On the SMIT platform, the contents are presented in a highly enriched way. In addition to the 
topic video, presented in Figure 2, learners can reach alternative videos, written materials, 
presentations, and infographics from the section on the right. In addition, they can perform 
learning tasks and learn the topic in depth by going to the "notes to the curious" section. 
Moreover, learners can take notes while working on the content and view the indicators related 
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to the topic from the right column. Learners can view all the topics in the course and the course- 
related dashboard from the left column. 


RESULTS 


In this study, learning analytics dashboards (LADs) were designed gradually according to 
different stakeholders' opinions. In this section, the development of dashboard elements as well 
as the findings of the three-phase data collection process are presented. 


3.1. Findings on learner expectations 


In order to determine learner expectations from the learning analytics, a web-based form 
consisting of an inclusive open-ended question is presented. Students stated a maximum 3 
expectations via this form. Consequently, the learners stated 36 information in total. However, 
not all of these expectations were considered during the design phase. Some of these are: 


P2: “T couldn't understand the learning task, can you explain in detail?” 
P22: “What should I do to be the best of the month??” 
P1: “Can you provide material to maintain the permanence of what I have learned?” 


Since every expectation recommended by the learners is not suitable for the features of the 
current system, nearly half of them were not used in the design phase. While the first comment 
above is about the feature of providing a more detailed explanation for the presented learning 
task, the second comment is about gamification elements and the third comment is about 
adaptively providing alternative content that already exists in the system to the learner. Within 
the scope of this study, since it is aimed to design LADs for an existing system, these comments 
were not evaluated during the design phase, since features that do not exist in the existing 
system cannot be added. Excluding these kinds of expectations, the remaining 19 information 
was analyzed in two categories as LA feedback and LA feedforward. 


LA feedback provides information about the gap between learners' current state and targeted 
state by utilizing learning analytics. LA feedforward provides a prediction based on learners' 
past learning experiences and/or the learning experiences of others. 9 of learner expectations 
were evaluated as LA feedback and 10 of them were evaluated as LA feedforward. 


3.2. Findings on design evaluation 


A draft design form was developed corresponding with students! expectations from learning 
analytics. This design form was first presented to the lecturers. Lecturers rated dashboard 
elements that might be useful for learners, and also defined new dashboard elements that might 
be meaningful for learners to encounter in the current system. The 24 dashboard elements that 
the lecturers agreed upon were presented to the learners for final evaluation. Of these 24 
dashboard elements, 15 contained LA feedback and 9 contained LA feedforward information. 
In addition, out of these 24 dashboard items, 15 were created based on learner expectations and 
9 based on lecturer recommendations. Examples of presented information in the dashboard 
element, depending on whether they are from lecturer recommendations or learner expectations 
and whether they include LA feedback or LA feedforward, are shown in Table 1. 


Table 1 Examples of LA feedback and LA feedforward 


LA feedback LA feedforward 
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Based on learner Trending material information Prediction of success in the topic 


expectations Performance trend information according to system interactions 
relative to others Information on when to complete the 
course 
Based on lecturer Mastery test performance Drop-out prediction information 
recommendations information Next topic estimation 


Log-in numbers information 


The learner views mentioned in the previous sections reflected the learners' expectations from 
learning analytics. Some of these learners (n=7) also participated in the design evaluation phase 
(third phase). The design evaluation form, which was developed according to the expectations 
of the learners at the first phase, was revised as a result of the lecturers’ opinions. The revised 
design form was presented to these learners upon their design evaluation. 


Unlike the form evaluated by the lecturers, the learners only rated scope and sample 
visualization of the dashboard element from | to 5. While the primary purpose of this phase is 
to provide the learner's views on the more concrete design, the secondary purpose is to identify 
the best among out of this information, since presenting 24 elements in the dashboard may 
result in negative learning outcomes. Hence, the elements that all 7 learners who participated 
in the research indicated 4 or 5 degrees out of 5 were included in the dashboard final design. 
Learners marked as 4 or 5 out of 5 for 12 elements out of 24 elements. Of these 12 dashboard 
elements, 8 were treated as LA feedback (later 3 of them revised as a single dashboard element), 
and 4 as LA feedforward. 


3.3. Dashboard elements 


In this section, the dashboard elements created as a result different stakeholder opinion will be 
introduced gradually. Since 3 of the agreed 12 dashboard elements (LA feedback) are 
composed of related metrics, they are graphed as a single dashboard element. Therefore, of the 
10 dashboard elements, 6 were LA feedback and 4 were LA feedforward. As a result, these 10 
dashboard elements obtained by the researchers were integrated into the system on two separate 
pages as topic-related and course-related. 


LADs are designed for MOOCs, which was developed from a project. There are many courses 
in MOOC platforms and each course has a syllabus in general. Each element of the syllabuses 
is treated as a topic. Therefore, the dashboard design in this study is configured separately as 
both course-related and topic-related. 
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Course-related dashboard 


The course-related dashboard can be viewed by 
clicking on this tab just above the topics. 


roe 
s 
= 
S 
° 
3 
pOegewson 


Botimsoll statistik z istatistik 


dersi tamamlayabilirsin. 


While course-related dashboard has 6 
such elements, topic-related dashboard 
has 4 elements. 


View of dashboard elements in 
course-related dashboard 


Fig 3. Screenshot of the course-related dashboard 


Figure 3 shows a screenshot from the course-related dashboard. Topic-related dashboard is 
designed similarly, and dashboard elements on both pages are discussed separately in this 
section. 


3.3.1. Course-related dashboard elements 


As a result of the study, 6 (4 LA feedback, 2 LA feedforward) dashboard elements are 
determined and located under the course-related dashboard. In this section, each element to be 
included in the course-related dashboard, stakeholder opinions referring to this element, the 
metrics required for the information to be included in the element, the calculation methods, and 
visualization of these metrics will be presented. 


Course-related element 1: Performance displays based on mastery test indicators 
This dashboard element covers 3 different information and presents mastery testing 


performances to learners descriptively. Therefore, the information presented here is considered 
as LA feedback. The illustration and explanations of this element are given in Figure 4. 
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What is presented with this element? 


Number of topics that learners are master/nonmaster 
or have not yet taken the mastery test 


Genel Bagan Performans: 


Ders Tamamlama Durumu 


The number of questions learners 
answered and correct response 
How much of the course rates in mastery tests. 


did learner complete? 
Fig 4. Course-related dashboard element-1 


The dashboard element was created by consolidating 3 different information. While 2 
information were recommended by the lecturers, | information was added based on the 
comment below. 


P15: “How much progress have I made in this course according to interactions on the 
system?” 


This dashboard element does not require any complex calculations. From this descriptive 
information, master/non-master/not taken metrics are presented as they are. Similarly, the 
number of questions encountered was obtained by adding the questions encountered in the 
mastery test in all topics, while the correct answer rate obtained by dividing the questions 
answered correctly by the number of questions encountered. The information in the progress 
bar under the element is obtained by dividing the topics of competence into the total topics. 


Course-related element 2: Norm-referenced feedback and self-referenced feedback according 
to recent performance 


Through this dashboard, the recent mastery testing performance of the learners is presented in 
comparison to the group and personal performance. The illustration and explanations for this 
element are shown in Figure 5. 
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What is presented with this element? 


Norm-reference feedback and self-referenced 
feedback according to recent performance 


Yetkiniik Gidigat Bagisi 
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otee 


The ten most recent Correct response rates 
mastery tests learners take 


Fig 5. Course-related dashboard element-2 


While the expectations of the learners were in the direction of norm-referenced feedback, self- 
reference feedback information was included in the design by the researchers. The comments 
of the learners for the related dashboard element are presented below. 


P3: “I am working but am I right? How is my performance compared to others?” 
P12: “Hey genie, how am I compared to other users?” 


As it was deduced from the comments, the learners wanted to see the performance compared 
to the others. This performance information was created according to mastery testing. The last 
10 mastery tests that the learner took and the correct answer rates are presented. Norm- 
referenced feedback is provided depending on three conditions. For these conditions, the 
standard deviation of the student's performance is defined. If it is greater than 1, determined as 
above the average, it is above the -1 determined as below the average and other conditions 
determined as at the average level. 


Self-referenced feedback compares the student's performance on the last two tests to the prior 
three tests. If 2 of the last 2 test performances are lower than the average of the previous 3 test 
performances, it is reported that the trend is negative, and if it is higher, the progress is positive. 
Otherwise, the message that the status is stable. 


Course-related element 3: Success predictions for all topics 


Through this element, it is aimed to provide a success prediction for all the topics. As it turns 
out, this element is covered under LA feedforward. The illustration and explanations of this 
element are shown in Figure 6. 
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What is presented with this element? 


Orange indicates the probability of success in the relevant topic, red indicates the 
probability of failure, and green indicates that you have already succeeded in that topic. 


Konu Bagan Kestirimieri > x 


Tum Konulara lliskin Yetkinlik Kestirimi 


© Yorkin Otscdejs = @ Yerkin Otrnamma Otauhis = @ Yerkin 


With this element learners can see at a glance; 

in which topics the learners are competent, 

with which topics the learners interact more (the more interaction in a topic, the more 
likely they are to be successful), 

which subjects are relatively difficult (the harder the subjects, the less likely they will be 
competent in that topic). 


Fig 6. Course-related dashboard element-3 


This dashboard element was created based on the expectations of 2 learners. In essence, learner 
expectations indirectly reflect this information. The comments of the learners for the related 
dashboard element are presented below. 


P3: “I have done so much right in this topic; will I be successful in the other topic or should I 
leave it here?” 


d 


P13: “Genie can tell my shortcomings and tell me the topics I need to focus on.’ 


As it mentioned in the comments, learners will have information about difficulty levels of the 
topics and the probability of being successful in topics according to their performance in a 
particular topic. In order to make this estimation, the correct response rate on all topics, the 
number of learning task views, and the number of course-related dashboard views were used. 
In addition, the duration of each topic-specific content, the number of video interactions 
(rewind, replay, note-taking) and the number of views on alternative content (pdf, pptx, 
infographic) were evaluated. Naive Bayes algorithm was used for predictions. 


Course-related element 4: Course completion time prediction 


In this dashboard element, the estimation on when the learner will complete the topic was 
presented based on their current performance. Thus, this element was evaluated under LA 
feedforward. The illustrations and the explanations related to this element are presented in 
Figure 7. 
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What is presented with this element? 


This LA feedforward provides the system prediction of how many days the course will 
be completed based on pace of progress and activity level. 


Dersi tamamlamana iligkin sistemin Ong6rusd 


Sistem, bu sekilde devam edersen e008 
2 2 U 


| a 


sonra dersi tamamlayacagin! 
Ongoriyor. 


“The system predicts that if you continue in the same way, you will complete the 
course after 22 days." 


Fig 7. Course-related dashboard element-4 
The learner expectations related to this element is given below. 


P14: “According to the information I have before the course, Genie can predict how long it 
will take to complete the course.” 


For this dashboard element learner expectations were revised based on current performance 
although previously learner expectations were based on prior learnings. The estimation 
regarding completion time was estimated utilizing metrics that are obtained by the time spent 
on mastery topic in the system, the time elapsed from the two component topics to the previous 
component topic, and the number of completed topics. k nearest neighbor algorithm was 
utilized in order to gather this estimation. 


Course-related element 5: Concepts learned 


In this dashboard element, learners are presented with the concepts they have learned according 
to the content of the topics they are competent in. This dashboard element is considered as LA 
feedback since it provides information regarding previous performance. The illustrations and 
the explanations related to this element are presented in Figure 8. 
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What is presented with this element? 


The concepts learned by the students are presented according to the 
topics they have mastered. 
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The more a concept is used in topics in which learners are 
master, the bigger it appears. 


Fig 8. Course-related dashboard element-5 
The learner expectations related to this dashboard element is given below. 


P22: “So, what I do know?” 


In order to create the word cloud, each topic and related concepts were paired based on 
instructor opinions. When the learner is competent in a particular topic, those concepts are 
added to the word with their weights. 


Course-related element 6: Trending material types 


The materials that learners are interested in the system are presented through this element. This 
dashboard element is considered as LA feedback. The illustrations and the explanations related 
to this element are presented in Figure 9. 
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What is presented with this element? 


Types of materials that learners can 
choose while studying on the topic 


Ogrenme Materyali Kullanim Dazeyleri > x 
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The blue lines represent the number of times the learner viewed the materials, while the black 
lines represent the average number of times the students enrolled viewed the materials. 


Fig 9. Course-related dashboard element-6 
The learner expectations related to this element is given below. 
P22: “What type of content did I tend to most?” 


The material information that learners as well as others are interested is presented. The number 
of existing videos, alternative videos, presentations, pdfs, and infographic elements are given. 


3.3.2. Topic-related dashboard elements 


4 of the dashboard elements developed as a result of the study (2 LA feedback, 2 LA 
feedforward) can differ according to the topics. Therefore, this dashboard is not in the main 
menus of the system, but in the content of a topic and is customized according to the current 
topic. 


Topic-related element 1: Roadmap of successful learners 


Through this element, learners can view how successful people follow their topic content. With 
this element, the message "you can be successful if you follow a path like this" is presented 
secretly. Therefore, this dashboard element is also evaluated under the LA feedforward. The 
notation and explanations of this element are shown in Figure 10. 
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What is presented with this element? 


X Konusunda Basarilt Olaniarin Etkilesim Oruntileri (Nereden Nereye Gitmisler?) 


Basarili 6Grenenlerin konu icindeki gezinimleri basarli 
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at 
Components that the learner can 
interact with within the topic 


When the learner hovers over a component, learner can see where successful learners are 
most headed from that component. 


For example, it can be seen that successful learners click on alternative content after the videos 
and then click on the course-related dashboard. 
Fig 10. Topic-related dashboard element-1 
The learner expectation associated with this dashboard element is presented below. 
P1: “What material will be most effective for me in my next learning?” 


Transition matrices have been created for this element. For example, information such as how 
many times successful learners have passed from the topic videos to the learning task on the 
relevant topic was displayed when hovering over the relevant component. 


Topic-related element 2: Time spent on the topic 


Through this element, the time spent by the learners is presented descriptively. Similarly, 
information about the average time spent by successful learners in the topic is presented. This 
dashboard element has been evaluated as LA feedback. The notation and explanations of this 
element are presented in Figure 11. 
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What is presented with this element? 


“Time you spent on this 
topic” 


X Konusunda Kalma Sireleri 
Bu konuda gecirdigin sire 


Bu konuda basarili olaniarin gecirdigi sure @ ~ 


59 dk ‘3 


"Time spent by those who are 
successful in this topic" 


Fig 11. Topic-related dashboard element-2 
The learner expectations associated with this dashboard element are presented below. 


P1: “How much time do I need to learn the next topic effectively?” 
P12: “What is the average time I have to spend for each topic?” 
For this dashboard element, the time spent by the learners on the relevant topic and the average 
time spent by the successful ones are directly presented. 
Topic-related element 3: Success prediction for the relevant topic 


Through this element, the topic estimation in course-related 3 is presented under each topic- 
related dashboard. This display format is indicated in Figure 12. 
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What is presented with this element? 


Probability of being successful in the current 
topic based on their interactions 


X Konusunda Basar Kestirimi 


%AS 


Bu sekilde devam edersen bu konuda basarili olma olasiligin 


"This is the probability of success 
if you continue in the same way" 


Fig 12. Topic-related dashboard element-3 
The learner expectations associated with this dashboard element are presented below. 


P2: “Can I be successful if I don't watch the video and only study the topic via pdf?” 


P17: “Hey Genie, I only worked with infographics and presentations. Will I be able to succeed 
in this topic?” 


With the LA feedforward in Course-related 3, the calculations and the algorithm used are the 
same, and the representation of this information is different. 
Topic-related element 4: Interactions for the relevant topic 


In this section material interaction levels of the learners after taking the proficiency test on a 
topic are demonstrated. This information was considered as LA feedback. The illustrations and 
the explanations regarding this element are shown in Figure 13. 
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What is presented with this element? 


The message presented according to the 
success of the learners in the topic 


X Konusunda OGrenme Materyali Etkilesimleri 


Tebrikler! Bu konuyu basariyla tamamladin! 


Bu konudaki iceriklerle etkilesim durumun asagida sunulmustur. 


7) infografik goriintiileme 7) Sunu goriintiileme 
x) Alternatif videolardan izleme 7) Video izlerken not alma 
7) Metinsel icerik gériintiileme 'x<) Merakhsina notlar inceleme 


The types of materials that the 
learner looked at in this topic 


Fig 13. Topic-related dashboard element-4 
The learner expectations related to this element is given below. 


P17: “I interacted with content on many topics. What content do you think I am more 
successful?” 


Learners can follow their level of mastery/non-mastery and their interaction with the related 
material. Hence, material interaction after taking the proficiency test can be followed by the 
learners. 


In sum, 10 dashboard elements were generated as a result of a systematic design process. These 
elements were developed under course-related and topic-related sections. Four elements on the 
course-related dashboard were LA feedback and two of them were LA feedforward. While on 
the topic-related dashboard three elements were LA feedback and one element was LA 
feedforward. The design process can be revised based on application and evaluation. However, 
the ultimate aim of this study is to reveal a systematic design process as well as the dashboard 
design based on this process. Future studies will contribute for development and optimization 
of this design. 
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4. DISCUSSION and CONCLUSION 


As a result of the systematic design process, it was determined that the expectations of students 
and lecturers from the learning dashboard were gathered under two main themes. First of all, 
it was agreed that the learning dashboard should be presented to visualize the current status of 
the students using the system. LA feedbacks, in which students' current situations are revealed 
depending on the system usage, are included in the learning dashboard. On the other hand, the 
future behavior and performance of the students has been tried to predict based on their current 
system usage behaviors and performances. These learning analytics indicators are also included 
in the learning dashboard as a LA feedforward. 


The dashboard elements to be presented at the end of a systematic data collection and design 
process are structured on two separate pages as course-related dashboard and topic-related 
dashboard. MOOCs platforms are generally competency-based and have a modular structure. 
Therefore, the topic-related dashboard presented here is an example for other MOOC 
platforms. 


As a result of the study, 10 dashboard elements were created, some of which consist of a single 
metric and some of which consist of many metrics. 6 of them were evaluated as LA feedback. 
The following information is presented with the dashboard elements evaluated as LA based 
feedback: 


e Performance displays based on mastery test indicators 

e Norm-referenced feedback and self-referenced feedback according to recent 
performance 

Concepts learned 

Trending material types 

Time spent on the topic 

Interactions for the relevant topic 


Dashboard elements evaluated as LA feedforward are as follows: 


e Success predictions for all topics 

e Course completion time prediction 

e Roadmap of successful learners 

e Success prediction for the relevant topic 

When researches on learning analytics are examined, most of the existing research until recent 
years included learning analytics reports based on descriptive statistics, aiming to determine 
the current behavior of students in the learning management system (Clow, 2013; Bakharia et 
al., 2016). Researchers emphasize that the benefit of reporting student data on LADs of learning 
management systems will be limited. The importance of interpreting these reports in a way that 
students can understand, making predictions about the future behavior and performance of the 
students, and making recommendations for students is emphasized (Jivet, Scheffel, Drachsler, 
& Specht, 2017; Jivet, Scheffel, Specht, & Drachsler, 2018). The importance of learning 
analytics indicators to be used for feedforward purposes, which will be included in LADs, 
becomes evident. In the study carried out by Leavy and Rheinschmidt (2010), a metric was 
developed to predict the success of the students in the online course. It was observed that there 
was a Significant increase in the success of the students who took the course as a result of the 
lecturers following the student based on this metric and making interventions when necessary. 
Similarly, it has been demonstrated that learning analytics has the ability to accurately predict 
risky behaviors and improve students' learning performance and learning outcomes (Du, Yang, 
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Shelton, Hung, & Zhang, (2021); Herodotou, Rienties, Hlosta, Boroowa, Mangafa, & Zdrahal, 
2020; Mubarak, Cao, & Ahmed, 2021; Namoun & Alshanqiti, 2021). 


It is stated that in the studies carried out on learning analytics indicators, the opinions of 
students and instructors are mostly not consulted on which metrics should be used. Researchers 
especially emphasize the importance of student-centered learning analytics, which also 
includes student and instructor opinions (Muljana & Luo, 2021; Ochoa & Wise, 2021). With 
this research, learning analytics indicators that were decided to be utilized as feedback and 
feedforward in the learning analytics dashboard were determined based on student and lecturer 
opinions. For future research, obtaining data on the performance of this learning panel, 
determining the improvement process of the learning panel, and the effect of panel usage on 
the learning process and results of the students. 
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Abstract 


Many scholars, educational institutions, and governmental bodies, including the National 
Research Council, consider science literacy and the Nature of Science understanding 
fundamental goals of K-12 STEM education. However, there is little research exploring the 
enhancement of Nature of Science understanding through theatre productions related to the 
history of science. This action research design proposal suggests that the adaptation and 
production of Life of Galileo by Bertolt Brecht could improve high school students’ online 
research skills, enhance students’ understanding of research, the scientific method, and the 
criteria that differentiate scientific from non-scientific theories. All the above improvements of 
skills could lead to the enhancement of students’ Nature of Science understanding. 


Introduction 
The Research Question 


Could the high school production of the theatre play Life of Galileo by Bertolt Brecht 
enhance the students’ Nature of Science understanding? 


Nature of Science in K-12 STEM Education 


The arguments for incorporating History and Philosophy of Science (HPS) (Klopfer & 
Cooley, 1963) in STEM (Science, Technology, Engineering, & Mathematics) education, the calls 
for science literacy (Rutherford & Ahlgren, 1989), and proposals for ways of implementing the 
teaching of Nature of Science (NOS) understanding (Lederman & Lederman, 2004) in 
educational institutions are not new. NOS is the general term defining the scientific enterprise 
that incorporates both HPS and science literacy. NOS is a fertile hybrid arena that combines 
elements of various social science studies. NOS incorporates history, sociology, and philosophy 
of science, coupled with cognitive science research, such as psychology, into a rich explanation 
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of what science is, how it operates, how scientists function as a social community, and how 
society itself both guides and reacts to the scientific goal (McComas, 1998, p. 4). 

Rutherford (2001), Teixeira, Greca, & Freire (2009), Garik & Benétreau-Dupin (2014), 
Matthews (2014, 2017, 2018), Michel & Neumann (2016), and other scholars argue that HPS can 
contribute to contemporary U.S. Science teaching for K-16 education and advocate the inclusion 
of HPS in their science syllabi (Matthews, 2014). Garik & Benétreau-Dupin (2014) argue that it 
is of high importance that students recognize that argumentation, criticism, and analysis are 
fundamental to science and that students should be educated to value science as part of our 
culture. Additionally, many scholars claim that the inclusion of NOS in STEM curricula would 
improve the content learning of students (McComas, 1998; Rutherford, 2001; Dass, 2005; 
Teixeira et al., 2009; Abd-El-Khalick, 2012; Garik & Benétreau-Dupin, 2014; Matthews, 2014; 
Michel & Neumann, 2016; Pellegrino, 2016; Gandolfi, 2019). 

According to Matthews (2014), many governmental and educational bodies in the past 
few decades have proposed the inclusion of HPS or NOS to their science syllabi, among which is 
the U.S. National Research Council (NRC), with its Next Generation Science Standards (NGSS) 
(National Research Council, 2013). Matthews (2014) argues that incorporating HPS into STEM 
curricula “has a significant contribution to make to improving science teaching and learning and, 
consequently, personal and social flourishing.” (p. 14). 

The justification of the NRC (2013) for integrating NOS into STEM education is that 
people need to know and understand the world around them. Moreover, humans need to modify 
their world by using technology to match what they understand or want. In certain situations, the 
need to know emerges from fulfilling essential needs in the face of possible risk. It is often a 
natural curiosity, and in other instances, it promises a better, more comfortable existence. 
Science is the pursuit of natural world explanations, while technology and engineering are the 
means to fulfill human needs, intellectual interests, and ambitions. Another consistent 
characteristic of scientific knowledge across disciplines is that scientific knowledge itself is 
subject to revision in the light of new evidence (p. 96). A scientifically literate person who can 
appreciate scientific knowledge’s essence is a fundamental aim of K-12 science education. 

Garik and Benétreau-Dupin (2014) drew the critical conclusion that science education 
must be integrated into a liberal arts education to prepare students to be equal participants in a 
participatory democracy. “Science teachers alone should not be expected to prepare students to 
be scientifically literate, and a modern curriculum that is organized through the humanities 
would be needed to educate students for scientific literacy.” (p. 1853). 

This action research design’s suggested learning process considers and integrates all the 
NOS factors of McComas’ (1998) description and follows the NGSS on NOS (NGSS, NRC, 
2013). The resources and tools suggested for this project to improve NOS understanding are 
debating and argumentation, online research, interdisciplinarity, and using art in STEM 
education, in this case, the performing arts, and more specifically, theatre. 


Using Theatre to Enhance Students’ Nature of Science Understanding 
The Drama-based pedagogy (DBP) is well researched and applied (Lee et al., 2015). It is 
a collection of drama-based teaching and learning strategies to facilitate learning in non-drama 


and drama-specific content through an embodied process-oriented approach to learning. 
Nevertheless, in DBP, the non-drama content focuses mainly on reading comprehension, social 


547 


and emotional skills, or other non-scientific areas. Also, DBP interventions are usually limited in 
their scope and duration. 

There are cases where theatre is implemented for teaching the philosophy of science 
(Toonders, Verhoeff, & Zwart, 2016)), popularizing the theory of evolution in a science museum 
setting (Peleg & Baram-Tsabari, 2016), or even dramatize original research and present it to 
elementary school students (Burgin, Alonzo, & Hill, 2016). Nevertheless, researchers have 
highlighted the limited use of drama in education. And while according to Toonders, Verhoeff, 
& Zwart (2016), “Drama is a relatively unexplored tool in academic science education,” Braund 
(2015) goes so far as to ask the question of whether drama and learning science is “an empty 
space.” 

Theatre has indeed been used to support science teaching (Giliberti et al., 2019; Abed, 
2016; Kerby et al., 2010; Peleg & Baram-Tsabari, 2011; Yoon, 2006) or history of science 
(Jansson & Aksela, 2013), but to a minimal degree. Some researchers have recently employed 
theatre to expand student understanding of NOS (Burke, Wessels, & McAvella, 2018; Burke, 
McAvella, & Wessels, 2020) or even train new science teachers on the NOS (Melo & Bachtold, 
2018). However, they were limited both in time and their scope. 


Adapting and Producing the Play Life of Galileo, by Bertolt Brecht 


This action research design is about the adaptation and production of the play Life of 
Galileo by Bertolt Brecht in the context of an elective course (e.g., Theatre I to Theatre IV 
Honors) offered to all high school students of a school in the Miami-Dade area in Florida. The 
book used for the production will be Life of Galileo (Student Edition) by Bertolt Brecht (Brecht, 
2020). The play is about the Italian natural philosopher’s career whom the Roman Catholic 
Church tried for the proclamation of his scientific discoveries. The play’s primary theme is the 
conflict between science and dogmatism. The students’ understanding of the experiment, the 
scientific method, and the criteria that differentiate scientific from non-scientific theories could 
be expanded through different stages of the play’s production. It is essential that students 
acknowledge that debating through valid arguments, criticism, and analysis, are fundamental 
operations in science and learn to value science as part of our shared human culture. The students 
will be evaluated through the readings, online research, rehearsals for the play’s production, and 
their replies to an open-ended questionnaire and interview administered three times during the 
school year. 

Lastly, this design will invite teachers from other disciplines to get involved and facilitate 
the students learning of the NOS, e.g., educators that teach language, literature, history, 
philosophy, social sciences, and STEM subjects, e.g., physics, mathematics, and biology. 


The Use of Online Research 


This research proposal intends to ask students, throughout all the stages of the play’s 
production, to conduct online research in small teams about the topic in question and decipher 
whether the sources they find are reliable scientific sources or not. Following the online research, 
each team could be asked to present and explain the topic to the rest of the class. In other 
instances, the whole class could be split into two parts and debate, using their argumentation 
skills, taking opposing views on the given topic. The topics could be assigned by the theatre 
teacher or emerge throughout the production processes of the play. The topics that emerge could 
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be specific to the historical events described in the play, for example, Galileo’s life, the 
telescope, the gravitational theories, e.t.c., or broader topics related to science. These broader 
topics could be, for example, the history and philosophy of science, the scientific method, and 
the history, the physics, or the mathematics of gravitational theories from Aristotle to Einstein. 


The Interdisciplinarity of the Project 


An essential aspect of the scientific endeavor is its interdisciplinary and collaborative 
nature. Understanding NOS includes the understanding that scientific inquiry is a process where 
a multitude of disciplines are contributing through the ages in the effort of the human species to 
understand nature and human’s place in it. Although the play is concentrated on Galileo, it is 
clear that people from other crafts and disciplines, for example, people like Giordano Bruno, 
Copernicus, a lens grinder, the inventor of the telescope, mathematicians, astronomers, 
philosophers, and other scholars, appear or are mentioned in the play (Brecht, 2020). Through all 
the production processes of the play, students could realize how the various disciplines are 
included, each of them offering their perspective and knowledge, providing a more multifaceted 
understanding of the scientific inquiry. 

Teachers from other disciplines will be invited as guest speakers to facilitate the 
conversation related to their field of expertise. The rationale for this is to highlight the 
interdisciplinarity of scientific inquiry and the collaboration of different scientific disciplines in 
scientific research. 

A language teacher could be a valuable resource as she can explain unknown words or 
scientific terms and help with the play’s language analysis, which will also accommodate diverse 
learners. The students analyzing the language used in the play and the words and expressions 
used in the 17th century (and are not used anymore) could significantly improve their 
understanding of the language of the play, the innuendos, and the nuances that would be missed 
without the language teacher’s input. The engagement of the English teacher might be essential 
for the first stage of the readings of the play. 

A literature teacher could work with the theatre teacher at the play’s analysis stage and 
the conversation about playwright Bertolt Brecht and his ideas. Furthermore, a literature teacher 
could help the theatre teacher explain the characteristics of Brecht’s Epic Theatre, a theatrical 
movement of the early 20th century based on theories and practices responding to the era’s 
political climate, leading to the creation of new political dramas. 

A history teacher will be needed for both the history of the Copernican revolution and the 
scientific revolution, together with their socio-cultural aspects. The theatre teacher has a master’s 
degree in history and philosophy of science and technology, and together with the history 
teacher, could navigate the students through the historical events that lead to Galileo’s 
discoveries and to the conflict with the Catholic Church, as well as the events that followed 
Galileo’s era. Students’ understanding of the history of science could enhance their NOS 
understanding. 

A philosophy teacher, in collaboration with the theatre teacher, will define what 
philosophy of science is and shed light on the philosophical aspects not only of the scientific 
revolution and the scientific method but also of the theological positions in opposition to the 
scientific conclusions of Galileo about the revolution of the Earth around the Sun. 

A physics teacher could offer more detailed explanations and insights into Galileo’s 
theory from a scientific perspective. The physics teacher could construct two experiments. The 
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first could be about measuring the acceleration of gravity using only a measuring tape to measure 
the altitude from which a ball is let fall and a stopwatch to measure the time it takes the ball to 
reach the floor. For the second experiment, the students could also calculate the acceleration of 
gravity using, this time, a swinging pendulum. The students could be informed that in all 
measurements, there are errors introduced. So, after repeating the experiments several times, the 
students will be taught how to use linear regression to find an approximation of the acceleration 
of gravity. A conversation could come after the experiments where the team will discuss the 
processes of physicists, the unavoidable errors introduced in measurement, and the reasons for 
recreating and repeating experiments. 

The mathematics teacher will work together with the theatre teacher (who also has a 
bachelor’s degree in mathematics and teaches mathematics at the school) concerning the 
mathematics involved in Galileo’s theory. Furthermore, a discussion could follow on the nature 
of mathematics as a model for physics and other sciences. To increase NOS understanding and 
accurately describe the scientific method, students need to understand the importance that the 
mathematical models of the natural phenomena in question play in creating a hypothesis or, 
eventually, a theory in physics. 

The biology teacher could also be proved essential because Galileo lived through two 
Plague outbreaks during his lifetime. A Plague outbreak takes place in one of the play’s scenes 
that affects the characters’ actions. In addition, given the COVID-19 pandemic and the 
requirement that students wear masks, an understanding of the biology aspect of pandemics is 
crucial not only for the analysis of the play and the understanding of the current situation but also 
for further deepening the NOS understanding of the students. 

The presence and interaction of all the teachers from different disciplines working 
together to help students understand the multitude of ways available to explore the nature of the 
scientific endeavor could result in students’ deepening of understanding of NOS. 


Assessment Instruments for NOS Understanding 


There are multiple assessment instruments for evaluating students’ NOS understanding 
that have been proposed, administered, researched, and validated (Lederman et al., 2002; 2004; 
2014; 2017, Lovelace & Brickman, 2013). Lederman et al. (2017) describe a plethora of NOS 
assessment instruments developed to assess students from 1954 to 2006 (Lederman et al., 2017, 
pp. 981-986). VNOS differs from typical paper and pencil instruments because of its open-ended 
nature. The results of their various studies and the follow-up interviews supported a high 
confidence level about the validity of the VNOS for assessing NOS understanding (p. 517). A 
historical review of the VNOS questionnaire and interview script can be found in the paper by 
Ayala-Villamil & Garcia-Martinez (2021). 

Lederman et al.’s paper (2017) is focused on evaluating the development of the open- 
ended instrument, Views of NOS Questionnaire (VNOS). The authors find evidence regarding 
VNOS validity for the range of the NOS aspects it aims to assess. Lederman et al. claim that 
establishing an instrument’s validity is always an ongoing process. The best the researchers can 
do is provide evidence of an instrument’s efficacy in measuring what it is designed to measure. 
The results of their various studies and the follow-up interviews supported a high confidence 
level about the validity of the VNOS instrument for assessing NOS understanding. 

For this action research project, we plan to use the VNOS-Form C instrument (Lederman 
et al., 2002, p. 509), adapting and improving it to assess students’ understanding of NOS before 
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the beginning of the classes, during the online research and rehearsals, and after the 
performances. 


The Timeframe of the Design Proposal 
First Grading Period 


Part of the classes at the beginning of the school year is the administration of the VNOS 
Form C Open-ended Questionnaire and Interview Script (Lederman, 2002, p. 509). The plan for 
the improvisations, theatrical games, and team-building activities is the themes to be taken from 
historical debates within the scientific community and between scientists and philosophers, 
religion advocates, or proponents of other disciplines of inquiry. Usually, during the first grading 
period, the students read the play, analyze the plot and the characters, and improvise on specific 
characters for the casting process. The students will be asked to conduct online research in teams 
about topics related to their characters, the era, the themes introduced in the play, and unknown 
scientific terms. Then they could be asked to explain the researched topic to the rest of the team 
or engage in a debate. The language, the physics, and the biology teacher could come as guests 
during this first grading period. All the guests will present their topics and then, with the help 
and the interventions of the theatre teacher will lead a discussion with the students. 


Second Grading Period 


By the beginning of the second grading period, the casting will be finalized. At this point, 
the students usually present parts of their scenes and explore their characters by going more in- 
depth in understanding their intentions, the relations with the other characters, and the plot. The 
students will be asked to research online and discuss what natural philosophy, the experiment, 
and natural philosophy’s methods were in Galileo’s era. The literature teacher could come as a 
guest to discuss Brecht and Epic Theatre with the students. The theatre teacher could present and 
discuss with the students the history of science from Mesopotamia (3000 BCE) to Italy (1600 
CE) and the history of optics in art and science, the camera obscura, and the telescope. The 
mathematics teacher could present the mathematics of Galileo’s theory and the process of 
creating a mathematical model of a natural phenomenon. The history teacher could present to the 
students the history of the Catholic Church and the Inquisition. The VWOS Form C Open-ended 
Questionnaire and the interviews are planned to be administered again at the end of this second 
grading period. 


Third Grading Period 


During the third grading period, the students learn their lines, improvise their scenes 
staying in character, and have more detailed conversations about their characters. During this 
period, it is suggested to discuss how science and other inquiry disciplines affect and form 
different opinions, psychological traits, and characters and how society affects the object of 
scientific inquiry and its direction. How cognitive science research is conducted, and how 
science 1s different from art. The philosophy and the social studies teachers could be invited 
during this grading period. These two teachers and the theatre teacher could suggest topics for 
students’ continuing online research, discussion, and debate. 
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Fourth Grading Period 


The fourth grading period is devoted to rehearsing whole acts, the dress rehearsals, and 
the performances. More elaborate conversations on Galileo’s scientific theory and the Catholic 
Church’s beliefs and why they oppose each other are proposed to happen from a philosophical 
point of view. Themes that could be discussed are the differences between scientific knowledge, 
beliefs, and opinions and how the same scientific data can have different interpretations within 
the scientific community. The conversations for the last grading period are suggested to be about 
NOS, HPS, and Science literacy. This last grading period is the time for the students to self- 
reflect on what they have learned about what science is and its relationship with society. The 
VNOS Form C Open-ended Questionnaire and the interviews are proposed to be administered 
once more before the end of the fourth grading period. 


Coda 


The author understands the theatre team as a community that is building its knowledge 
collectively. The theatre teacher is a part of that community. Student questions about different 
themes, topics, or terms that could emerge at any point of the production process of the play 
could change the suggested timeframe of this proposal. Following a predetermined schedule 
independently of the flow, the process, and the level of student understanding defies the purpose 
and the iterative nature of action research. Action research, both as a philosophy and 
methodology of research, is a process that leads to transformative change through the iterative 
operation of taking action and doing research, linked and becoming a whole through critical 
reflection (Stringer, 2014). 


Data Collection and Methodology 


Throughout the school year, the qualitative data intended to be collected are observations 
and field notes, videos of all the lessons and the interviews, three answered VNOS Form C Open- 
ended Questionnaires from each participant student, and the interviews of the students. The 
assessment instrument for measuring the students’ NOS understanding will be the Views of 
Nature of Science - Form C Questionnaire and Interview Script, which can be found in Views of 
Nature of Science Questionnaire: Towards Valid and Meaningful Assessment of Learners’ 
Conceptions of Nature of Science (Lederman et al., 2002). At the end of the school year, all the 
observation and field notes, the videos, the three answered questionnaires per participating 
student, and the transcripts of their follow-up interviews will be put together for the qualitative 
data analysis that will follow. The answers to the questionnaires and the transcribed interview 
texts will be examined and interpreted to understand what they represent. This coding process of 
labeling and organizing the qualitative data will identify different themes and their relationships. 


Views of Nature of Science - Form C Questionnaire and Interview Script 
The Views of Nature of Science - Form C Questionnaire and Interview Script (Lederman 


et al., 2002, p. 509) comprises ten open-ended questions. Based on the questionnaire and the 
students’ answers, the researcher will interview the students to ask clarification or follow-up 


552 


questions using the ten questions and the students’ replies as the interview script. The ten open- 
ended questions are seen below. 


1. What in your view is science? What makes science (or a scientific discipline such as 
physics, biology, etc.) different from other disciplines of inquiry (e.g., religion, philosophy)? 


2. What is an experiment? 


3. Does the development of scientific knowledge require experiments? 
* If yes, explain why. Give an example to defend your position. 
¢ If no, explain why. Give an example to defend your position. 


4. After scientists have developed a scientific theory (e.g., atomic theory, evolution 
theory), does the theory ever change? 

¢ If you believe that scientific theories do not change, explain why. Defend your answer 
with examples. 

* If you believe that scientific theories do change: (a) Explain why theories change? (b) 
Explained why we 

bother to learn scientific theories? Defend your answer with examples. 


5. Is there a difference between a scientific theory and a scientific law? Illustrate your 
answer with an example. 


6. Science textbooks often represent the atom as a central nucleus composed of protons 
(positively charged particles) and neutrons (neutral particles) with electrons (negatively charged 
particles) orbiting that nucleus. How certain are scientists about the structure of the atom? What 
specific evidence do you think scientists used to determine what an atom looks like? 


7. Science textbooks often define a species as a group of organisms that share similar 
characteristics and can interbreed with one another to produce fertile offspring. How certain are 
scientists about their characterization of what a species is? What specific evidence do you think 
scientists used to determine what the species is? 


8. It is believed that about 65 million years ago the dinosaurs became extinct. Of the 
hypotheses formulated by scientists to explain the extinction, two enjoy wide support. The first, 
formulated by one group of scientists, subjects that a huge meteorite hit the Earth 65 million 
years ago and led to a series of events that caused the extinction. The second hypothesis, 
formulated by another group of scientists, suggests that massive and violent volcanic eruptions 
were responsible for the extinction. How are these different conclusions possible if scientists in 
both groups have access to and use the same set of data to derive their conclusions? 


9. Some claim that science is infused with social and cultural values. That is, science 
reflects the social and political values, philosophical assumptions, and intellectual norms of the 
culture in which it is practiced. Others claim that science is universal. That is, science transcends 
national and cultural boundaries and is not affected by social, political, and philosophical values, 
and intellectual norms of the culture in which it is practiced. 
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* If you believe that science reflects social and cultural values, explain why. Defend your 
answer with 

examples. 

* If you believe that science is universal, explain why. Defend your answer with 
examples. 


10. Scientists perform experiments/investigations when trying to find answers to the 
questions they put forth. Do scientists use their creativity and imagination during their 
investigations? 

* If yes, then at which stages of the investigations you believe scientists use their 
imagination and creativity: 

planning and design, data collection, after data collection? Please explain why scientists 
use imagination 

and creativity. Provide examples if appropriate. 

* If you believe that scientists do not use imagination and creativity, please explain why. 
Provide examples if 

appropriate. 


Expected Outcomes 


A critical reflection could inform and adapt the elective Theatre course based on the 
implementation of this action research design and its conclusions. Through this process of 
analysis of learning needs and goals, the end product of this design is suggested to be developing 
a delivery system to meet the students’ needs in successfully enhancing NOS understanding 
through the school production of theatrical plays relative to the history of science. 


References 


Abd-El-Khalick, F. (2012). Teaching with and about nature of science, and science teacher 
knowledge domains. Science & Education, 22(9), 2087-2107. https://doi- 
org.access.library.miami.edu/10.1007/s11191-012-9520-2 

Abed, O. H. (2016). Drama-Based Science Teaching and Its Effect on Students’ Understanding 
of Scientific Concepts and Their Attitudes towards Science Learning. /nternational 
education studies, 9(10), 163-173. 

Ayala-Villamil, L. A., & Garcia-Martinez, A. (2021). VNOS: A Historical Review of an 
Instrument on the Nature of Science. /nterdisciplinary Journal of Environmental and 
Science Education, 17(2), e2238. 

Braund, M. (2015). Drama and learning science: An empty space? British Educational Research 
Journal, 41(1), 102-121. 

Brecht, B. (2020). Life of Galileo (Student Edition). Methuen Drama. 

Burgin, S. R., Alonzo, J., & Hill, V. J. (2016). Dramatizing the Authentic Research of a Local 
Scientist to Urban Elementary Students Through Professional Theater. Science & 
Education, 25(9), 1073-1088. 

Burke, L. E. C.-A., Wessels, A., & McAvella, A. (2018). Using Theater and Drama to Expose 
and Expand the Epistemic Insights of Youth Regarding the Nature of Science. Research 


554 


in Science Education (Australasian Science Education Research Association), 48(6), 
1151-1169. 

Burke, L. E. C.-A., Wessels, A., & McAvella, A. (2020). Using Drama to Uncover and Expand 
Student Understanding of the Nature of Science. The Science Teacher (National Science 
Teachers Association), 88(2), 28. 

Dass, P. M. (2005). Understanding the Nature of Scientific Enterprise (NOSE) through a 
discourse with its history: The influence of an undergraduate ‘History of Science’ 
Course. International Journal of Science and Mathematics Education, 3(1), 87-115. 
https://doi-org.access. library.miami.edu/10.1007/s10763-004-3225-1 

Gandolfi, H. E. (2019). Exploring and implementing an intercultural model of history of science 
to teach about nature of science (Doctoral dissertation, UCL (University College 
London). 

Garik, P., & Benétreau-Dupin, Y. (2014). Report on a Boston University conference December 
7-8, 2012 on How can the history and philosophy of science contribute to contemporary 
USS. science teaching? Science & Education, 23(9), 1853-1873. 

Giliberti, M., Adorno, D. P., Pizzolato, N., Battaglia, O. R., & Fazio, C. (2019, August). 
Augmented lectures: benefits of supporting physics teaching with the theatre. In Journal 
of Physics: Conference Series (Vol. 1286, No. 1, p. 012065). IOP Publishing. 

Jansson, J., & Aksela, M. (2013). Science theatre in teaching and popularizing the history of 
chemistry: The case of a theatrical play on Nobel-laureate A.I. Virtanen. Lumat: 
International Journal of Math, Science and Technology Education, 1(4), 457-475. 

Kerby, H. W., Cantor, J., Weiland, M., Babiarz, C., & Kerby, A. W. (2010). Fusion Science 
Theater Presents The Amazing Chemical Circus: A New Model of Outreach That Uses 
Theater To Engage Children in Learning. Journal of Chemical Education, 87(10), 1024- 
1030. 

Klopfer, L., & Cooley, W. (1963). The history of science cases for high schools in the 
development of student understanding of science and scientists. Journal of Research in 
Science Teaching, 1(1), 33-47. 

Lederman, N. G., Abd-El-Khalick, F., Bell, R. L., & Schwartz, R. S. (2002). Views of nature of 
science questionnaire: Towards valid and meaningful assessment of learners’ conceptions 
of nature of science. Journal of Research in Science Teaching, 39(6), 497-521. 

Lederman, N., & Lederman, J. (2004). Revising instruction to teach nature of science: Modifying 
activities to enhance students’ understanding of science. The Science Teacher, 71(9), 36- 
39. 

Lederman, J. S., Lederman, N. G., Bartos, S. A., Bartels, S. L., Meyer, A. A., & Schwartz, R. S. 
(2014). Meaningful assessment of learners’ understandings about scientific inquiry—The 
views about scientific inquiry (VASI) questionnaire. Journal of research in science 
teaching, 51(1), 65-83. 

Lederman, N. G., Bartos, S. A., & Lederman, J. (2017). The development, use, and interpretation 
of nature of science assessments. In M. R. Matthews (Ed.), International handbook of 
research in history, philosophy and science teaching (pp. 971-997). Springer. 

Lee, B. K., Patall, E. A., Cawthon, S. W., & Steingut, R. R. (2015). The effect of drama-based 
pedagogy on preK—16 outcomes: A meta-analysis of research from 1985 to 2012. Review 
of Educational Research, 85(1), 3-49. 

Lovelace, M., & Brickman, P. (2013). Best practices for measuring students’ attitudes toward 
learning science. CBE—Life Sciences Education, 12(4), 606-617. 


555 


McComas, W. (Ed.). (1998). The nature of science in science education: Rationales and 
strategies. Kluwer Academic Publishers. 

Matthews, M. R. (2014). Science teaching: the contribution of history and philosophy of science, 
20th anniversary revised and expanded edition. Rutledge. 

Matthews, M. R. (2017). International handbook of research in history, philosophy and science 
teaching. Springer. 

Matthews, M. R. (Ed.). (2018). History, philosophy and science teaching: New perspectives. 
Springer. 

Melo, E., & Bachtold, M. (2018). A Theater-Based Device for Training Teachers on the Nature 
of Science. Science & Education, 27(9), 963-986. 

National Research Council. (2013). Next generation science standards: for states, by states. The 
National Academies Press. https://www.nap.edu/catalog/18290/next-generation-science- 
standards-for-states-by-states 

Peleg, R., & Baram-Tsabari, A. (2011). Atom Surprise: Using Theatre in Primary Science 
Education. Journal of Science Education and Technology, 20(5), 508-524. 
http://dx.doi.org/10.1007/s10956-011-9299-y 

Peleg, R, & Baram-Tsabari, A. (2016). Understanding Producers’ Intentions and Viewers’ 
Learning Outcomes in a Science Museum Theater Play on Evolution. Research in 
Science Education, 46(5), 715-741. 

Pellegrino, J. W. (2016). 2/st century science assessment: The Future Is Now. SRI International. 

Rutherford, F.J., & Ahlgren, A. (1989). Science for all Americans. Oxford University Press, Inc. 

Rutherford, F. J. (2001). Fostering the history of science in American science education. Science 
Education and Culture, 41-52. https://doi.org/10.1007/978-94-010-0730-6_ 4 

Stringer, E. T. (2014). Action research in education. Pearson. 

Teixeira, E. S., Greca, I. M., & Freire, O. (2009). The history and philosophy of science in 
physics teaching: A research synthesis of didactic interventions. Science & 

Education, 21(6), 771-796. https://doi.org/10.1007/s11191-009-9217-3 

Toonders, W, Verhoeff, R.P., & Zwart, H.A.E. (2016). Performing the future: On the use of 
drama in philosophy courses for science students. Science & Education, 25(7), 869-895. 

Yoon, H. G. (2006, May). The nature of science drama in science education. In the 9th 
international conference on public communication of science and technology (PCST-9). 
Seoul, Korea. 


556 


Exploring Levels and Patterns of Social Presence in Asynchronous Online 
Discussion (AODs): A Longitudinal Study 


Qi Wu 
School of Education, Syracuse University 
259 Huntington Hall 
Syracuse, NY 13244 


qwul6@syr.edu 


Tiffany A. Koszalka 
School of Education, Syracuse University 
264 Huntington Hall 
Syracuse, NY 13244 
takoszal@syr.edu 


Zhijuan Niu 
School of Education, Syracuse University 
259 Huntington Hall 
Syracuse, NY 13244 


zhniu@syr.edu 


Abstract 


The prevalence of shifting face-to-face classes to online learning under the pandemic suggests 
that social presence is crucial in overcoming inherently impersonal characteristics of online 
instruction. Observation techniques, social network mapping, and content analysis, using the 
Community of Inquiry framework (Col), were conducted in a longitudinal study exploring patterns 
of social presence across two online courses over two semesters for a single cohort of graduate 
students. Patterns and suggestions for the instructional design are discussed. 


Keywords: Asynchronous Online Discussions; Social Presence 
Introduction 


Education largely depends on social interaction and effective communication (Lowenthal & 
Dunlap, 2018). Interaction has been stated as one of the critical aspects of the online learning 
environment (Moore, 1989; Wagner, 1994) since it enables learners to gain other people's 
perspectives through interaction, and it is essential to create a learning community advocated by 
Garrison, and other learning theorists who emphasize the crucial role of community in learning 
(Garrison, Anderson & Archer, 2000). Social presence and interaction are closely related (Huang, 
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Gan, Wen & Li, 2017), that the concept of presence has been used to understand interactions in 
online learning environments (Saadatmand et al., 2017). Arbaugh et al. (2008) suggest that the Col 
is a powerful and relevant theoretical framework to examine and explain online learning 
effectiveness and provides opportunities for researchers to evaluate learners' interaction and 
experiences in online learning environments. In Community of Inquiry framework (Col), social 
presence is a popular construct used to describe and understand how people socially interact in 
online learning environments (Whiteside, Dikkers, & Swan, 2017). Garrison (2009) defined social 
presence as "the ability of participants to identify with the community (e.g., course of study), 
communicate purposefully in a trusting environment, and develop interpersonal relationships 
progressively by projecting their individual personalities." 


An appropriate pedagogical design and proper exploitation of social technologies are crucial 
in fostering the processes of communication and interaction (Saadatmand et al., 2017). 
Asynchronous online discussions (AODs) are among the most widely used instructional 
techniques to support students' online learning (Koszalka, Pavlov & Wu, 2021; Gao, Zhang & 
Franklin, 2013). Participating in AODs through sharing thoughts, asking questions, and providing 
feedback, students are provided with ample possibilities that promote interaction and 
communication, thus building community in the online learning environment (Yang, Yeh & Wong, 
2010). 


Considerable effort has been devoted to studying social presence's establishment, 
notwithstanding, the nature and development of social presence in the online learning environment 
is still an ongoing issue (Lowenthal & Dunlap, 2018, 2020; Rourke & Kanuka, 2009; Swan & 
Shih, 2005). The majority of the social presence research sought to evaluate and measure students' 
perceptions of social presence by solely employing self-report surveys and interview instruments, 
which overlooked the importance of assessing and observing students’ actual social presence 
behaviors with specific indicators. Empirical evidence also suggests that students’ perceptions 
derived from self-reporting surveys do not accurately represent and truly match with their actual 
behaviors and participations in online courses (Picciano, 2002). Bernard et al. (2009) reported in 
their meta-analysis on interaction that research on interaction in distance education and online 
learning tends to focus more on interaction interventions (i.e., conditions or environments designed 
to elicit interaction) than on students' actual interactions (i.e., their behaviors). In addition, many 
of the previous social presence studies were conducted as short-term studies within the same 
context with different participants. Given the importance of social interaction in terms of 
developing social presence, it is necessary to further analyze how students interact online to 
establish social presence, thereby achieving an interactive and communicative online learning 
community that is meaningful to their learning experiences. 


Beyond the self-report surveying generally used in this type of research, this longitudinal study 
explored the patterns and levels of the same cohort of graduate students' social presence in AODs 
across two online courses over two semesters. We sought to explore various patterns among 
students' social interactions and relationships in AODs. Observation techniques, document 
analysis, and social network mapping were employed to provide another point of analysis that 
faithfully characterizes the characteristics and the development of social interactions and social 
presence in the online environment. 
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Research Questions 


Based on previous research limitations that justify the need for conducting the current study, 
the questions guiding this study focus on the nature and development of social presence. Typically, 
social presence is established and maintained in AODs through social interactions among the same 
cohort of students over the long term across two semesters. The specific research questions were: 


1. What are the patterns of social presence within a single cohort of students in AODs 
across two online courses over two semesters? 


2. What are the levels of social presence within a single cohort of students in AODs across 
two online courses over two semesters? 


Method 

Participants & Context 

The study participants were the same cohort of twelve full-time graduate students who enrolled 
in two consecutive online courses over two semesters (Fall 2020 & Spring 2021) for an MS 
Instructional Design at a private northeastern university in the United States. The average age of 
the participants was 44 years old, and most of the participants were male (80%). The majority of 
the participants were active duty (80%), and the rest were veterans (20%). The average number of 
previous online courses have taken among participants was above four. 


Data Collection & Analysis 

Data collected from AOD scripts from two different consecutive online courses was under an 
IRB exempt status. The analysis comprises embedded case studies, 5 cases of AODs in Fall 2020 
course, and 5 in Spring 2021. Transcripts of postings from two courses were retrieved from the 
Blackboard LMS, downloaded, cleaned, and saved in the qualitative data analysis software 
MAXQDA before coding. Transcript analysis was applied using the categorical indicators defined 
in the Col framework, where social presence was analyzed in the transcripts by coding for affective 
responses, interactive responses, and cohesive responses (Swan, 2003; Hughes et al., 2007). The 
unit of analysis was sentence, whereby a single sentence could include multiple social presence 
indicators. Through an iterative coding practice process, two coders finalized the coding scheme 
with an inter-rater agreement of 0.75. Social network mapping analysis was conducted through 
Gephi 0.9.2 program to investigate and visualize the participation patterns and interaction levels 
manifested within and across the two courses over two semesters. 


Results 

Social Presence Category Pattern 

The descriptive data for the Fall 2020 course revealed that cohesive responses were present the 
most and had the highest overall mean scores (M= 98.6), followed by interactive (M= 90.2) and 
affective responses (M= 37.6). Across all the AODs in the Fall term, there was a relatively minor 
mean score difference between interactive and cohesive responses (Minteractive= 99.2; Mconesive= 
98.6) than the mean score differences between these two categories and the affective responses 
category (Myf fective= 37.6) since it was the only category had a mean score below 50 across two 
courses. This suggested that throughout the AODs in the Fall term, students barely used words or 
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sentences that indicate their self-projection and acceptations of others into and within the learning 
community (Rourke et al., 1999). 


Though the frequency and the mean score orders of social presence categories in the Spring 
2021 course remained as the same pattern as the Fall 2020 course, the sharp decrease in the overall 
interactive responses and the moderate increase in the affective responses resulted in a larger 
difference between interactive and cohesive responses (Minteractive= 76.0; Mconhesive= 101.2) 
comparing to the one that the Fall term had. The overall increase in affective responses from the 
Fall term (M= 37.6) to the Spring term (M= 51.4) within the same cohort of students suggested 
that providing opportunities for longer interaction time and collaboration experiences might help 
facilitate student degree of comfort in recognizing each other in an online learning community. 


Social Presence Indicator Pattern 

Filling the gap of the majority of previous social presence research studies in the literature that 
“did not report results at the indicator level” (Lowenthal et al., 2020), this study looked at and 
compared the occurrence and frequency of individual social presence indicators across two courses. 
As the findings suggested, vocatives (Mrqi= 46.8; Mspring= 49.6) and group references (Mrqy= 
42.8; Mspring= 49) were two mostly used social presence indicators that had the highest mean 
scores across both courses, followed by complimenting, expressing appreciation (Mpqj)= 34.8; 
Mspring= 31), and acknowledgment (Mrqy= 25.8; Mspring= 25.4). On the contrary, embracing the 
group (Mpqu= 3.6; Msnring= 9) was the least frequently used indicator, followed by humor (Mpqu= 
4; Mspring= 4-2), and greetings and salutations (Mrqy= 5.4; Mspring= 2-6). 


Overall, across two courses, the increasing trend in the affective responses (e.g., self-disclosure, 
paralanguage & humor) and the decreasing trend in interactive responses (e.g., 
agreement/disagreement, invitation, complimenting, and expressing appreciation) suggested that 
students felt and became more comfortable in sharing personal experiences with their peers as they 
keep collaborating from the Fall term to the Spring term, but tended to less focusing on exchanging 
meaningful knowledge or ideas related to the course content. 


Student Interaction and Relationship Patterns Across Two Courses 

Social network analysis was conducted to map out the differences of the same cohort of 
students’ interactive activities patterns and relationships between two courses as shown in Figure 
1, where the Fall 2020 course is on the left, and the Spring 2021 course is on the right side. Each 
map includes 12 nodes representing a student and one node representing the instructor (INS). In 
the visualization, the node size indicates each student’s level of activity in responding to others’ 
posts in AODs. It was calculated by the total number of responses that the student sent out divided 
by the total number of participants. The node color indicates students’ popularity and centrality 
within the AODs calculated by the total number of responses the student received divided by the 
total number of participants. The higher the number, the more popular the student is. The edge and 
its width connecting the nodes represent the frequencies of participant interactions, and the arrows 
point out the directions of interactions. 


560 


Fall 2020 Spring 2021 


Figure 1. /nteraction Patterns and Relationships in Two Courses 


By comparing the social network maps between the two courses, there were more nodes in 
dark green in the Spring 2021 course than in the Fall 2020 course. This suggested that students 
replied as well as received more responses among each other in the Spring term than in the Fall 
term. Overall, the node sizes in the Spring 2021 course were relatively larger than those in the Fall 
2020 course, which also demonstrated a higher student participation rate in replying to others’ 
posts. There was an increasing number of thick edges among students in the Spring 2021 course 
than in the Fall 2020 course. Most of the thick edges among students in the Spring 2021 turned to 
be more stable and evenly distributed, which suggested that the Spring semester course had created 
a more trusting and sociable interactive learning community, where students had expanded their 
communications without a fixed social circle after experiencing previous AODs in the Fall 
semester. In addition, there was an increase in the number of students who had popular postings 
in the Spring course that aroused resonations and sympathies among peers. Though the sizes of 
instructor’s nodes in both courses were almost the same, the Spring 2021’s instructor had a much 
darker green node, which suggested the instructor’s posts or comments had received more 
attentions among students as well as elicited more discussions with students. 


Levels of Social Presence within Each AOD Across Two Courses 

The percentage value for each of the social presence categories in each AOD case across two 
courses were calculated based on the number of coded sentences that contained one social presence 
category in one certain case divided by the sum of coded sentences that containing that particular 
category within all five cases in each course. The two courses shared the same highest level of 
affective responses and cohesive responses (28%) and the similar highest level of interactive 
responses (Fall = 25%; Spring = 26%). The majority of the thread sentences were coded as 
Cohesive Responses (44% in both Fall 2020 & Spring 2021), indicating a steady level of group 
cohesion within the same cohort of graduates over two courses. 


Overall, both courses experienced similar observable fluctuation patterns of social presence 
level, which suggested that different time points throughout the course semester might influence 
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students’ level of participation and engagement in interacting with each other. Besides, different 
types of discussion topics and questions might also influence the level of social presence. 


Discussion 

Instruction Stages 

Spending time in the Fall 2020 semester to interact and learn through participating AODs 
activity, the same cohort of students established and developed social presence continuously and 
intensely in the Spring 2021 semester. This study showed that different stages of AODs activity 
throughout the semester had an essential impact on student social presence level. The same 
repeated patterns of social presence level across two courses suggested that during the late half of 
the semester, students might experience course fatigue, which will lead to a lack of interest and 
motivations in participating and engaging in AODs. Specifically, the observable fluctuations were 
in the affective and cohesive responses. Therefore, this informed online instructional designers 
and educators to carefully plan and arrange AOD activity based on different stages to effectively 
sustain student efforts and volitions in interacting with each other as well as to avoid having a 
sense of loneliness and lack of interest in participating in AODs due to the inherently impersonal 
characteristics of the online learning environment. 


Instructor Involvement 

Social presence, however, may not be increased over time without appropriate instructional 
design (Akyol et al., 2011; Shea, 2006). Students may not perceive a higher social presence in a 
long-term course if the instructor does not provide more interaction opportunities and proper 
instructional approaches (Lee et al., 2018). Research has reported that the level of instructor 
involvement has an impact on student social presence. Lowenthal et al. (2020) found low instructor 
involvement helps build social presence, and findings in An et al.'s (2009) study showed students 
tended to express thoughts and opinions more freely with many social presence cues when the 
instructor’s intervention was minimal. In this study, both courses had a limited level of instructor 
involvement, as shown in Figure 1. Nonetheless, there was still a higher level of social presence 
categories in the Spring 2021 course that could be attributed to the instructor’s role in AODs. As 
thus, while keeping a minimal level of instructor involvement to help establish a higher level of 
social presence, online instructors should also be attentive to other instructor role-related aspects 
that might influence student social presence, such as the time for instructor entering and posting 
messages, the quality of the discussion feedback given, and the tones and styles of asking prompt 
questions that might arouse further and deeper discussions. 


In this study, the decrease in interactive responses in the Spring 2021 course indicated that the 
same cohort of students related and shared more about personal experiences as they were getting 
used to interact and communicate with each other as a group, but less course content learning- 
related behaviors were noticed. Thus, this suggested instructor should make sure to keep students 
are interacting with each other in a meaningful learning way. Establishing social presence means 
creating a climate that supports critical inquiry to achieve educational outcomes (Garrison & Akyol, 
2013). According to the Col framework, the teacher plays an important role in building social 
presence. The instructor establishes relationships and a sense of belonging by designing, 
facilitating, and directing cognitive and social presences. The teacher’s personality, teaching styles, 
approaches, and beliefs influence the development of presence (Casey & Kroth, 2013). There is a 
lack of research focusing on the instructional effect on helping students build social presence in 
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the literature. Future research could examine whether there is a causal relationship between 
different instructional approaches and student social presence to investigate why students’ 
interactive responses decreased over time, and which instructional strategies could effectively 
facilitate a higher level of social presence in online courses. 


Previous relationships 

Research has suggested that having a past relationship with class members is helpful when 
establishing social presence in online courses, where Lowenthal et al. (2018; 2020) found a 
positive group project experience helps increase students’ perception of social presence and further 
helps maintain relationships with others. The results of this study confirmed their findings in a way 
that it is valuable that having “a cohort model enables students’ multiple opportunities to build 
relationships with others across semesters” (Lowenthal, 2020) since the Spring 2021 social 
network map showed a more stable and expanded scope of student interactions with each other. 
Moreover, since all the participants in this study had a military background, even they did not have 
previous social relationship, their similar background set up a learning community that is easy for 
them to understand each other. Future studies could examine social presence within a cohort of 
students with different backgrounds, such as military versus on-campus students. In addition, it 
would also be helpful to study students individually based on their characteristics. 


Conclusion 


This longitudinal study investigated the social presence among the same cohort of students 
across two courses. Student discussion posts indicated that social presence, as analyzed through 
observation of behaviors, was an important part of AODs. Findings in this study help unpack the 
complexity and establishment of social relationships and provide insights into making AODs more 
productive. Thus, inform instructional designers and online educators on possible techniques to 
enhance and evoke student participation and engagement in AODs to gain a better and meaningful 
online learning experience. Hence, the analysis of these case studies provided further insights into 
the growth of social presence levels in AODs. 
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Abstract 


The purpose of the study was to investigate college language faculty’s use of 
technologies to organize teaching and learning in language classrooms during emergency remote 
teaching necessitated by the pandemic in the spring semester of 2020. The study used Mishra and 
Koehler ‘s (2006) theoretical framework of the Technological Pedagogical Content Knowledge 
(TPACK) to analyze the five Chinese language instructors’ strategies to virtually engage their 
students. The results indicate three themes in the participants’ ERT relative to student 
engagement: Frequent communication, ERT-sustainable materials, and quick turnaround times 
for interaction. The study results offer theoretical and practical implications for both researchers 
and instructors. 


Keywords: TPACK, emergency remote teaching (ERT) 


Introduction 


Language classrooms are typically interactive because the instructor and students are 
often engaged in a variety of activities to practice the language skills, such as drills, instant 
feedback, role-plays, and discussions. These exercises usually work well with the participants’ 
physical presence. However, what would happen to language teaching and learning when a face- 
to-face spontaneity is absent from the instruction and when an exclusive reliance on technology 
as an instructional medium is necessary? The pandemic outbreak during the spring semester of 
2020 abruptly migrated most college faculty and students in the United States from an in-person 
instructional setting to an emergency remote teaching (ERT) environment. ERT in this study 
refers to a temporary shift in the teaching modality that extemporaneously transforms what 
would have otherwise been face-to-face teaching components into a digital form. 


One of the key differences that sets apart ERT from well-structured online instruction is 
its urgent, emergent nature. At many institutions, teaching online in a regular semester is a 
“bottom-up” affair (Lesht & Windes, 2011), in which faculty members elect to do so and have 
time for preparation. In addition, typically students are enrolled in web-based courses by choice. 
The willingness of faculty and students can be pivotal for successful technology integration into 
the instructional practice in non-emergency circumstances (Reid, 2012). Moreover, virtual 
teaching and learning involves instructors and learners in a process of role transformation 
(Cochran & Bemuto, 2016). Many online educators and students experience a shift in their roles 
from an in-classroom instructor and student with a visible center of attention to an invisible 
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facilitator and users of online materials (Bair &Bair, 2011). Such a transition is often a 
movement on a continuum, and the faculty and students continue to adjust themselves in their 
new roles and learn to navigate how they can better engage with one another and with the subject 
content as online instruction proceeds. That is, in a non-crisis online educational setting, both 
faculty and students can pace themselves in their practical and psychological preparation as they 
transition to online instruction. However, that is not the case for ERT, when the situation 
indiscriminately demands a rapid transition from faculty and students, including those who did 
not have the skills or resources needed to thrive in the digital landscape. While less than ideal, 
this underprepared transition in the teaching modality presented both pedagogical challenges and 
growth opportunities. 


The purpose of the study was to analyze college faculty’s use of technologies to organize 
teaching and learning in language classrooms during ERT. Guided by Mishra and Koehler ‘s 
(2006) theoretical framework of the Technological Pedagogical Content Knowledge (TPACK), 
the study attempts to answer the following research question: How did Chinese language faculty 
apply their technological pedagogical content knowledge to engage students during ERT? The 
research findings are important because they evaluated the technological pedagogical content 
knowledge of instructors, who were not pedagogical technology specialists but completed their 
ERT successfully. The evaluation offers insights on the theoretical and curricular role that 
TPACK plays in faculty development. Moreover, this study discussed student-content 
engagement, one core emergent theme in the data, and raised questions regarding sustainability 
of virtual teaching materials. 


Literature Review 


Mishra and Koehler (2006) assert that when using technology to facilitate teaching, 
teacher knowledge of technology is context-dependent and integral to the knowledge system of 
subject matter and pedagogy. The technology knowledge has “a ripple effect” (p.1025) affecting 
faculty’s instructional decisions because the use of technology, such as a Learning Management 
System (LMS), can change how instructors organize and then present the content information in 
a way that would have been different in the absence of technology. These technological 
pedagogical knowledge and skills need to be learned by faculty before effective application of 
educational technology can take place (Mohr & Shelton, 2017). However, in the reality, online 
instructors in a well-planned teaching context can still experience challenges. The faculty 
participants in Eichelberger and Leong’s study (2019) reported that technical failures caused 
students’ frustration and impatience with the instructors during their synchronous sessions. The 
participants in their study believed that these negative outcomes would have been mitigated had 
teachers and students had more specific knowledge to troubleshoot effectively. While some 
technical issues are evidently beyond faculty’s control (e.g., an internet outage), they still affect 
instructional productivity. This leads to an inquiry regarding what faculty can do to plan and 
foster positive online student engagement and offer both instructional and social support in terms 
that enhances the teaching outcomes that technological efforts alone or the lack thereof cannot 
achieve in a virtual learning environment. 


TPACK Framework 
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Teaching online is a complex cognitive activity that relies on teachers’ flexible access to 
their many organized knowledge systems fundamental to teaching. To offer grounding that 
describes interplay among essential qualities of teacher knowledge necessary for successful use 
of educational technology in their teaching, Mishra and Koehler (2006) proposed Technological 
Pedagogical Content Knowledge (TPACK) framework based on Shulman’s (1986) 
epistemological concept of pedagogical content knowledge with an integrative addition of 
technology. Shulman’s model considers pedagogical content knowledge as a qualifier that 
defines the profession of teaching: teachers consider the process of teaching and learning and 
then accordingly interpret, organize, represent, and formulate the subject matter in 
comprehensible ways for their students. That is, content is transformed for teaching through 
teachers’ application of pedagogical knowledge. Mishra and Koehler’s TPACK is 
conceptualized in a similar way with an added domain of technology knowledge, and the 
integrated technological pedagogical content knowledge can change educators’ consideration for 
not only instructional delivery but also the overall curriculum design (Voogt et al., 2011), such as 
offering online options in addition to in-person sessions. 


TPACK is commonly represented using a Venn diagram with three overlapping circles 
representing the three core types of knowledge: content, pedagogy, and technology. The model 
proposes that the interplay among the three circles results in four additional types of knowledge 
with technological pedagogical content knowledge sitting in the center (see Figure 1). 


Pedagogical Content 


Pedagogical 


Knowledge Content 


Knowledge 


Technological Content 
Knowledge 


Technological Pedagogical 
Knowledge 


Technological 
Knowledge 


Figure 1 The TPACK framework (Archambault & Barnett, 2010, p. 1954) 


The framework addresses three core interdependent constructs, three pairs of connected 
knowledge domains, and the centrally overlapped area. Pedagogical content knowledge is the 
integration of subject matter expertise and mastery of teaching skills. Technological content 
knowledge concerns the ways that application of technology changes the nature of content 
representation and learning processes. Technological pedagogical knowledge points to a 
teacher’s knowledge regarding the available technological tools and teachers’ capacity to 
strategically choose technologies to enhance learning outcomes. Coming together, technological 
pedagogical content knowledge is the basis of meaningful teaching with technology and requires 
faculty’s understanding of 
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the representation of concepts using technologies; pedagogical techniques that use 
technologies in constructive ways to teach content; knowledge of what makes concepts 
difficult or easy to learn and how technology can help redress some of the problems that 
students face; knowledge of students’ prior knowledge and theories of epistemology; and 
knowledge of how technologies can be used to build on existing knowledge and to 
develop new epistemologies or strengthen old ones (Mishra & Koehler, 2006, p.1029). 


The TPACK theoretical model is often used for analyzing teacher’s integrated knowledge 
of technology, pedagogy, and content and examining how teachers apply technologies to 
represent, formulate, and transform the subject so it is comprehensible to learners. However, 
teachers’ knowledge is complex, and an interconnectedness of multiple elements can make 
isolation and measurement of these individual types of knowledge challenging. Hence, while 
TPACK offers multiple categories of teachers’ knowledge relative to content, pedagogy, and 
technology, researchers have been challenged by the fuzzy boundaries between adjacent 
constructs that are uneasy to separate out (Graham, 2011). Moreover, TPACK does not fully 
examine contextual factors known to impact instruction, such as faculty beliefs about their online 
teaching using TPACK (Eichelberger & Leong, 2019). Nevertheless, TPACK is helpful in 
organizing teachers’ relevant knowledge domains (Archambault & Barnett, 2010). 


Student engagement 


Student engagement is crucial to improve student satisfaction, motivation to learn, sense 
of learning community, and academic performance (Martin & Bolliger, 2019). In understanding 
the importance of student engagement, Laurillard (2000) articulated that the purpose of higher 
education moves far above information access and aims to enable individuals to develop their 
capabilities to the fullest through “engagement with others in the gradual development of their 
personal understanding” (p.133). Such engagement, according to Anderson (2003), is developed 
through interaction among faculty, students, and subject content. In other words, both human and 
nonhuman engagement activities are essential and complementary elements of quality teaching 
and learning experiences. The research findings of Handelsman et al. (2005) suggested that an 
individual’s interactivities with their peers, instructor, and content leads to an engaged learner, 
and that engagement is a significant predictor for the student academic achievement. The study 
of Abdous and Yen (2010) also supported the significant role of learner-to-teacher interaction in 
learning outcomes and student satisfaction in an online environment. Moreover, the study of 
Lear et al. (2010) showed that social engagement fostered among peers and faculty through class 
interactivity develops a sense of community, and this sense of belonging contributes to student 
learning and places faculty at the center of community development. Quality teaching and 
learning is reciprocal, responsive, and social, not competitive or isolated, and engagement among 
faculty, students, and content affords the participants opportunities to exchange ideas and 
sharpen thinking (Chickering & Gamson, 1987). Hence, educational engagement involves both 
social and cognitive aspects in an online learning community. The construct of student 
engagement in this study is situated within the ERT parameters and therefore refers to the 
instructor’s use of technologies to organize their teaching activities and purposefully create 
virtual learning opportunities through students’ interaction with the instructor, their peers, and 
the subject content. 
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Methodology 
Participants 


With the approval from the institutional review board at the researcher’s university, a 
non-probability purposive sampling procedure was adopted to identify five faculty participants to 
represent the most diversity within the constraints (Teddlie & Tashakkori, 2008). A snowball 
recruitment technique was used to compile an initial list of twenty college Chinese language 
instructors who taught during ERT in the United States. Subsequently, the twenty instructors 
were cross-categorized by their genders, years of experience, position titles, and institutional 
types. One or two participants from each category were then selected to form the purposive 
sample. The five participants taught in four different classifications of institutions in the United 
States: state research-oriented, state teaching-focused, private research-emphasized, and private 
liberal arts. The participants were all native speakers of Chinese originated from either China or 
Taiwan between the ages of twenty-eight and sixty-five. The participants included both male and 
female, and their position titles ranked from Instructor I to Full Professor. 


Interview questionnaire instrument 


Voogt et al. (2013) pointed out that ultimately online instructors need to be competent in 
teaching their specific subject areas with technology and thus suggested that teacher’s use of 
technology be assessed using TPACK that addresses domain-specific knowledge. As such, the 
instrument specified the interconnectedness among technology, language pedagogy, and Chinese 
language. Guided by the theoretical TPACK framework and the research question, an open- 
ended, semi-structured interview instrument was used for data collection during the 
videoconferencing interviews. To increase the instrument content validity and questioning 
techniques, three college professors who taught Chinese language during ERT but were not 
participants in the study were recruited for the field test. The instrument was subsequently 
revised according to the three professors’ feedback. The five interviews ranged from sixty to 
eighty minutes, and they were centered on the following four core inquiries with varying follow- 
up questions based on the participants’ narratives. 


1. Can you share an overview of the differences in how you engaged your students before 
and during ERT? 

2. Technological knowledge: Please describe your experience in using educational 
technologies before and during ERT. 

3. Content knowledge: Please describe your specialized areas in Chinese language teaching 
and the materials you used before and during ERT. 

4. Pedagogical knowledge: Please describe your teaching rationale and how you 
purposefully presented the subject content and create an effective learning experience 
before and during ERT? 


Data analysis 


The data included the video recordings, field notes, and analytic memos. They were 
member checked and coded using Saldana’s (2009) the first and second coding procedure for 
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recursive themes and explanations. The initial round of coding sought to list major topics relative 
to TPACK within individual interviews. The second coding focused on the shared topics across 
the five interviews. The third round of coding analyzed the interconnectedness among sub- 
datasets within and across the interviews. The final coding synthesized the thematic sub-datasets 
and prepared for listing findings. To verify the validity of the findings, two colleagues, who 
participated in the field test, checked the final codebook for data consistency and repeatability 
(Trochim, 2006) and verified that the listed findings were supported by the data (Guion, 2002). 


Findings 


This study sought to answer the question: How did Chinese language faculty apply their 
technological pedagogical content knowledge to engage students during ERT? The participants 
shared that while they were neither specialized in educational technologies nor a regular user of 
technologies beyond emailing and creating presentation slides before ERT, they became able to 
proficiently apply new technologies to engage students in a variety of learning activities during 
ERT. Three types of engagement that were described across the participants during ERT are as 
follows. 

e Instructor-students interaction, such as intentional and frequent communication with 
students via voice messages for pronunciation assistance, routine check-in emails with 
Graphic Interchange Formats (GIFs) and emoji inserted, and one-on-one synchronous 
meetings to support both students’ academic and personal needs 

e Student-content interaction, such as the use of thought-provoking materials recorded by 
the students to invite sustainable virtual discussions 

e Student-student interaction, such as adoptions of discussion-board topics that allowed 
quick responses and short turnaround times 


One common theme in the discussions regarding the participants’ approaches to create an 
effective ERT experience was their use of the course suspension days to transition their classes 
into ERT. During the course suspension periods, which ranged from three to fourteen days 
depending on the institution, while the faculty were not allowed to assign students any academic 
work, the participants strategized the time to collect information about their students’ academic 
and personal needs. The participants used online free survey tools or connected with their 
students individually through email or videoconferencing to inquire about the students’ 
expectations of their learning during ERT. Based on the information, the faculty participants 
revised the syllabi accordingly. Participant One reported, “My students indicated in the survey 
that they wanted to finish the semester strong and were concerned that the learning outcomes 
would be compromised. As such, my goal during ERT was not only to survive but also to 
thrive.” The participants also communicated with their students using online discussion boards 
during the suspension periods to share questions and answers among the students. In this 
endeavor, the participants intended their information exchanges to be frequent, quick, and brief. 
The twofold intentions of frequent communication through varying modalities were reported to 
provide timely academic and psychological support during the learning interruption. Moreover, 
in creating an effective ERT experience, the participants needed to reorganize teaching materials 
in effective ways, such as digitalizing the materials and centralizing the location of the files on 
the LMS. Furthermore, considering the challenges associated with students’ personal lives (e.g., 
home environments not suitable for synchronous learning), the participants offered their students 
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frequent individualized assistance through supplementary text-based, recorded, and synchronized 
instruction so the students could achieve the learning outcomes as they would have had they 
remained on campus. 


The participants reported that the development of students’ spoken skills and grammar 
knowledge continued to be key instructional objectives during ERT. In explaining the 
implementation of different pedagogies during ERT, Participant Two shared that “Technological 
constraints affected my instructional clarity. Hence, I sent pronunciation corrections through 
voice messages alongside text explanations. I also recorded a series of grammar lectures that my 
students could watch outside the class hours and have more time and space to reflect on them.” 
Examples of technological constraints in this case are as follows. Pronunciation drills on 
videoconferencing were less effective due to the lack of eye contact and rhythm. Moreover, the 
instructor could not instantly pick up markers of different colors to enhance key points or 
handwrite Chinese characters on the screen easily. According to the student evaluation of 
teaching, the students found voice messages and videos helpful because they were engaged with 
the content through a familiar voice and pedagogy. 


In addition to increasing more time and space for students’ independent learning during 
ERT, the participants discussed the need for shortening turnaround times in a videoconferencing 
setting. Videoconferencing technology did not allow the instructor to attend to multiple breakout 
rooms simultaneously. Moreover, the transition between big and small groups often interrupted 
the class momentum. Upon realizing the technological limits, all participants chose not to 
continue virtual small group practice. As a result, when the entire class was present, only one 
student or one group could practice conversations at a time, and the rest had to just wait on the 
screen. This arrangement then created a new challenge, for large group provided less practice 
opportunities for each individual than small groups work. In responding to the technological 
challenges, keeping the class momentum, avoiding unmoderated breakout room activities, and 
involving more students to practice speaking during group learning, the participants compensated 
for the absence of small group work for smaller tasks. Specifically, Participant Three created a 
pool of varying smaller tasks for frequent teacher-student and student-student interactions by 
“downsizing” the regular speaking exercises, which would have otherwise been more extensive 
in a traditional classroom when physical contextual cues (e.g., body movements and acting) were 
available. Smaller-size language tasks using visual and audio prompts to elicit student output 
allowed quick turnaround times and enabled everyone in class to have new questions instead of 
repeating what had already been answered by their peers. Alongside the concept of quick 
turnaround times, the participants emphasized that the exchanges on the discussion boards 
featured frequent “short and sweet” responses to invite participation and move discussions along. 


In elaborating on ERT pedagogies, the participants further discussed the importance of 
meaningful student-content interaction. To engage students with the subject content in a 
comfortable pace during ERT, Participant Four designed the speaking exercises in an online 
collaborative space where students regularly recorded their verbal comments based on 
multimedia prompts, such as videos, images, and voice recordings. The instructor then offered 
timely feedback on the same collaborative space. According to Participant Four, these exercises 
could make up for small group discussions, which were reported to be ineffective in their 
beginning-level language instructional context through videoconferencing. Moreover, to sharpen 
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students’ critical thinking skills, sustain their interest in the discussion, and build students’ 
vocabulary, Participant Five purposefully asked students to videotape masking and COVID-19 
situations in their towns or countries to stimulate discussion on first-hand information. 
According to the participant, the students were enthusiastic, and the discussions were engaging, 
which pushed the students to use more expressive vocabulary words. Participant Two added that 
when the materials could cause reactions from students, they were “‘idle-time-free” materials that 
were especially valuable in an online classroom. 


Discussions 


Overall, the findings reported that the participants did not consider themselves as 
specialists in digital literacy over pedagogical technologies before ERT. Regardless, the 
participants did not encounter or perceive any technical difficulties that stopped them from 
engaging their students during ERT. The participants concluded that they achieved their teaching 
goals and finished ERT successfully. This finding supports Lee’s (2009) study that foreign 
language instructors who teach remotely are encouraged to “have good general computer literacy 
and skills, but they do not necessarily have to be familiar with each technology” (p. 250) in order 
to succeed in their teaching. Their digital literacy enabled the participants to implement an array 
of ERT instructional activities, and the three reoccurring pedagogical themes relative to student 
engagement during ERT were frequent communication, ERT-sustainable materials, and quick 
turnaround times for interaction. The data suggested that the participants purposefully selected a 
variety of technologies to help them to meet these pedagogical needs. Informed by the TPACK 
model, this section discusses how the participants presented effective instruction with technology 
and analyzed the teaching activities that would not have been accomplished without the 
integration of technology. 


Frequent communication 


Teacher-student engagement was reported to be more frequent. The high frequency 
occurred outside the instructional time, including both the suspension periods and throughout 
ERT, via online survey tools, email, and one-on-one videoconferencing. The increased frequency 
in communication was intentional and strategized by the participants to gather students’ input on 
how to best organize teaching, provide academic assistance, and offer student wellness support 
during the unprecedented interruptions. These purposeful, frequent communication activities 
with individual students (e.g. polling students’ learning preferences during ERT and promptly 
answering pronunciation questions via voice messages in addition to a written explanation) 
wouldn’t have been accomplished without the integration of technology. That is, to create 
varying representations of the subject content and achieve their teaching objectives, the faculty 
participants needed to deploy different types of technologies. These technological tools assisted 
the participants to include various communication genres and purposes, such as written and oral, 
visual, audio, and text, formal and informal, concise and elaborate, social-affective and 
educational, and individual and community-based. The participants’ use of technology showed 
their “ability to choose a tool based on its fitness, strategies for using the tool’s affordances, and 
knowledge of pedagogical strategies and the ability to apply those strategies for use of 
technologies” (Mishra & Koehler, 2006, p. 1028). These implementations are evidence of the 
participants’ technological content knowledge. 
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In addition to the increased occurrence of communication, the participants’ use of 
symbols in written communication also changed. Many emotions were derived from 
interlocutors’ physical presence (e.g., proximity, body movement, and posture), and because they 
could not be replicated during ERT, the participants inserted emojis, GIFs, and memes more 
frequently as a means of establishing animated engagement with their students. According to 
Resnick (1991) that instructor-student rapport can greatly affect the nature of student learning 
and educational outcomes. It is because when members in a learning community find the 
communication and interaction enjoyable and personally fulfilling, they are more likely to 
remain in the learning circle for the duration of the educational process and uphold the academic 
rigor (Garrison et al., 2000). As such, the frequent, positive connections with the students may 
have contributed to the participants’ achievement in attaining their pedagogical objectives during 
ERT. In this study, the participants first identified characteristics of communication during ERT, 
“the nature of the target audience” (Mishra & Koehler, 2006, p. 1027), and their pedagogical 
options. Subsequently, based on their understanding of the emergent instructional context, the 
participants chose technological tools that could easily poll student information, complement 
other tools (e.g., voice messages along with text explanations), and compensated for the loss of 
in-person communication during ERT. These teaching activities and decisions, which involved 
integration of the three core constructs of technology, pedagogy, and content, reflected the 
participants’ application of TPACK in their virtual instruction. 


ERT-sustainable materials 


Academic rigor remained crucial to both the participants and their students during ERT. 
As such, the theme of sustainability in material creation and reorganization was reoccurring in 
the data. The analyses showed two characteristics of the materials that qualified them to be 
sustainable in terms of enacting student-content engagement and providing academic support 
during ERT. First, thought-provoking was the feature that the participants were looking for when 
selecting videoconferencing-effective materials. This rationale is supported by the study results 
of Martin and Bolliger (2018) that online discussions on topics that encourage deep reflection to 
be the most valuable engagement strategy. Specifically, the “heated” discussions over the 
masking policies engaged Participant Five’s students with the content, and such pedagogically 
crafted learner-content interaction could introduce “changes in the learner's understanding, the 
learner's perspective, or the cognitive structures of the learner's mind” (Moore, 1993, p.20). 
Participant Five’s use of contentious videos, which were filmed by the students, about how 
COVID-19 masking was enforced in different areas was an example of the participant’s 
application of technological pedagogical content knowledge. This thought-provoking learning 
opportunity with every student meeting via videoconferencing in real-time but bringing first- 
hand materials from different physical spaces could not have happened had the students been on 
campus together or had the necessary technologies been absent. The physical distance between 
students in this case was central to Participant Five introducing unique teaching materials that 
actively maintained student interest on a flat screen. The physical distance also contributed to the 
student learning through collecting and using authentic materials. In this example, the participant 
used the knowledge of technology productively and considered both content and pedagogy 
concurrently when designing this well-timed activity to stimulate and sustain the virtual language 
practice through dynamic discussions. 
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Instructional clarity was the second feature that the participants aimed to achieve when 
creating ERT-sustainable teaching materials that students could resort to outside the synchronous 
instructional hours. Participant Two made brief teaching videos and make them accessible through 
a free video sharing website to supplement student learning. The goals were to enhance students’ 
comprehension of the subject matter and allow the students additional space and time to interact 
with the content asynchronously. The participant’s pedagogies in this regard are supported by 
Meskill and Anthony’s (2010) view that instructors of remote teaching should use text, sounds, 
and visuals to amplify instruction. Such instruction should also provide with diverse accessibilities 
and computer-mediated cues that can appeal to learners of varying learning styles and encourage 
them to engage with learning activities via student-computer interaction. The participant’s 
strategies to complement synchronous human interaction with reflective student-content 
interaction created a quality teaching and learning experience (Anderson, 2003). With the creation 
of the video lessons that supported student learning, Participant Two facilitated learning discourse 
by engaging the students in “interacting about and building upon the information provided in the 
course instructional materials” (Garrison et al., 2007, p. 164). The coming together of Participant 
Two’s knowledge in technology affordance, subject matter, alternative representations of concept 
brought an innovative, well-received outcome and shows an example of faculty application of 
TPACK to teaching. 


Quick turnaround times for interaction 


The data showed that inefficient implementation of concurrent multiple breakout rooms 
for group work through videoconferencing was a common concern. The participants redressed this 
limitation by creating a greater variety of quick-turnaround exercises for synchronous interactions, 
keeping asynchronous discussion board activities short and focused, and offering prompt 
responses to students’ recorded speeches in an online collaborative space. One feature shared in 
these pedagogical techniques is their quick turnaround times. Brevity and promptness in these 
teaching activities were intended to motivate and enable more individuals to participate in the 
exercises within the allotted time and space. In addition, the participants created varied 
representations of the subject matters using images, audio recordings, and videos to engage the 
students with the content through different stimuli. The choice of technologies afforded the varying 
types of content ideas for the faculty participants to enrich their teaching. Being able to navigate 
the spaces flexibly defined by the three components of content, pedagogy, and technology and the 
interplay among them in their specific contexts shows the participants’ integrated knowledge of 
TPACK. 


The examples aforementioned in this section involved participants knowing not just the 
subject matter but also “the manner in which the subject matter can be changed by the application 
of technology” (Mishra & Koehler, 2006, p. 1028). In addition, the participants demonstrated their 
knowledge of the existence and utilities of various technological tools and accomplished the 
delivery of their teaching virtually. These ERT experiences in relation to instructor-student, 
student-content, and student-student engagement would not have been possible without the 
integration of technologies. The faculty participants’ teaching techniques and technological 
choices were based on their pedagogical rationale and their understanding of the content. Together, 
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it was the participants’ discipline-specific TPACK knowledge that enabled them to enact “truly 
meaningful and deeply skilled teaching with technology (Koehler & Mishra, 2009, p.66). 


Conclusion 


The findings and implications must be interpreted within limitations. The study used a 
small, non-randomized sample. Hence, the results are not generalizable to a larger population. 
The interviews as the single data source also presents a limitation. Moreover, because the 
participants were asked to discuss their ERT experiences, potential recall errors and subjectivity 
are also limiting factors (Patton, 2001). Nevertheless, the qualitative study approach offers 
readers with a greater depth in analysis (Stake, 1995). The study results are significant in the 
following theoretical and practical implications. First, the participants of the study were 
competent in navigating the complex interactions among content, pedagogy, and technology as 
language teaching professionals who resorted to their general computer literacy and used varying 
technologies to organize their teaching rapidly during ERT. Their last-minute integration of 
technology into their pre-existing teaching practice was a success based on the participants’ 
perceptions and the institutional student evaluation of teaching. This experience seems to suggest 
a different approach to TPACK than Koehler and Mishra’s (2009) perspective that teachers and 
teacher educators should “move beyond oversimplified approaches that treat technology as an 
‘add-on’” (p. 67). In this study, technology was a component that joined the pre-existing 
teaching practice at a later stage during Spring 2020. The implication of this observation raises a 
question that if an “add-on” approach and general digital literacy suffice to lead to a positive 
result of faculty’s use of the TPACK model, how does this inform the decision of teacher 
education programs and institutional professional development programs in relation to their 
curricular component of pedagogical technology? The issue with material sustainability and 
effectiveness in a virtual language learning environment reoccurred in the data. The participants 
had solutions to the challenges with varied degrees of success that need to be investigated. 
Moreover, practitioner research is necessary to evaluate the discipline-specific demands on 
online learning materials and subsequently to assess their effects on both teaching and learning. 
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